Guidelines: The do's, don'ts and don't knows of direct observation of clinical skills in medical education

Educational utility of observational workplace-based assessment modalities in Australian vocational general practice training: a cross-sectional study

BACKGROUND

Direct observation, workplace-based assessments (WBAs) are a fundamental component of competency-based postgraduate medical education. In Australian general practice vocational training, external clinical teaching visits (ECTVs) are key observation-based WBAs. Traditionally, ECTVs are conducted face-to-face, but the COVID-19 pandemic saw the development and implementation of remote ECTV modalities. It remains unknown if perceived educational utility of remote ECTVs differs from traditional face-to-face ECTVs. This study explored the educational utility of ECTVs, including face-to-face and remote formats.

METHODS

General practice trainees ('registrars') and external clinical teaching visitors ('ECT visitors', who are independent experienced GP observers) each completed a cross-sectional questionnaire following individual ECTVs undertaken in 2020. Outcomes included overall educational utility of the ECTV as perceived by registrars, registrar ratings of likelihood to change their clinical practice as a result of the ECTV, registrar ratings of likelihood to change their approach to learning/training as a result of the ECTV, and overall educational utility of the ECTV as perceived by the ECT visitor. Educational utility ratings (5-point scales) were analysed descriptively. Univariable and multivariable logistic regression were employed to examine factors associated with dichotomised educational utility ratings.

RESULTS

Response rates were 41% (n = 801) for registrars and 39% (n = 742) for ECT visitors. Most registrars (64.1%) rated ECTV overall educational utility as 'very useful'; 58.5% and 47.9% of registrars rated their likelihood to change practice and approach to learning/training, respectively, as 'very likely'. No statistically significant differences in perceived educational utility ratings were identified between face-to-face and remote video/phone ECTVs (multivariable p-value range: .07-.96). Receiving feedback that was focused/specific/easy to translate into action was consistently associated with registrars' rating overall educational utility as 'very useful' (odds ratio (OR): 12.8, 95% confidence interval (CI): 8.26 to19.9), rating likelihood to change practice as 'very likely' (OR: 2.5, 95%CI: 1.59 to 3.94), and rating likelihood to change learning/training approach as 'very likely' (OR: 3.19, 95%CI: 1.97 to 5.17).

CONCLUSIONS

ECTVs are perceived by registrars and ECT visitors to be educationally useful across different delivery modalities and formats. The quality and features of the feedback provided appear most important in ECTVs as an assessment for learning.

The Impact of Instructional Approaches on Scaling and Root Planning Skill Learning in Preclinical Periodontal Education

OBJECTIVE

This study aimed to evaluate the effectiveness of video-based learning, live demonstrations, and brochures in teaching scaling and root planning (SRP) skills to third-year dental students in a preclinical periodontal education setting.

METHOD

Ninety third-year dental students who had not previously received theoretical or practical training in SRP were included in this randomized study. The students were equally assigned to three instructional groups: brochure, video, and live demonstration. To evaluate the initial effectiveness of these methods, all participants first performed SRP on standardized typodont models coated with simulated calculus, created using a consistent application of black nail polish. Each student performed 10 strokes per surface using identical instruments to ensure standardization. Pre- and post-procedure images of the models were taken for objective assessment. The extent of calculus removal was quantified through pixel area measurements using image analysis software. Following the practical sessions, a validated 10-item Likert-scale questionnaire was administered to assess students' perceptions of the instructional methods, focusing on clarity, engagement, satisfaction, and preferences for future learning.

RESULTS

Significant improvements in the quality of scaling and root planing were observed in all three groups after education. Video and live demonstration methods were significantly more effective than the brochure method. However, there were no significant differences between video and live demonstration groups. Student feedback showed that both the video and live demonstration methods received significantly higher scores than the brochure method; however, there was no significant difference between the video and live demonstration methods.

CONCLUSION

Both video-based learning and live demonstrations are equally effective and superior to brochures for teaching critical periodontal skills. These results suggest that interactive teaching methods should be prioritized in dental education to enhance student outcomes and better prepare them for clinical practice.

Gothenburg direct observation tool for assessing person-centred care (GDOT-PCC): evaluation of inter-rater reliability

OBJECTIVE

To assess the inter-rater reliability of the Gothenburg direct observation tool-person-centred care in assessing healthcare professionals' competency in delivering person-centred care (PCC).

DESIGN

Observational, fully-crossed inter-rater reliability study.

SETTING

The study was conducted between October and December 2022 at the participants' homes or offices.

PARTICIPANTS AND METHODS

Six health professionals individually rated 10 video-recorded, simulated consultations against the 53-item, 15-domain tool covering four major areas: PCC activities, clinician manner, clinician skills and PCC goals. Cronbach's α was used to assess internal consistency. Intraclass correlations (ICC) and 95% CI were computed for the domains.

RESULTS

Two domains (planning and documentation and documentation) were excluded from analyses due to insufficient evaluable data. Cronbach's α was acceptable (>0.70) for all evaluated domains. ICC values were high (ICC ≥0.75) for 11 of the 13 domains; however, CIs were generally wide and the lower bounds fell within the good range (ICC=0.60-0.74) for six domains and fair agreement (ICC=0.40-0.59) for the remaining six. The ICC for the domain patient perspective was non-informative due to its wide CIs (ICC=0.74 (0.39-0.92)).

CONCLUSION

ICC estimates for most domains were comparable to or exceeded those reported for similar direct observation tools for assessing PCC, suggesting that they may reliably be used in, for example, education and quality improvement applications. Reliability for the domains planning and documentation and documentation needs to be assessed in studies sampling more documentation behaviours. Reliability for the patient perspective domain may owe to methodological issues and should be reassessed in larger, better-designed studies.

Local clinical practice patterns in urolithiasis guidelines: a critical evaluation from Turkey

PURPOSE

This study aimed to evaluate the current clinical practice patterns regarding the utilization of "Urolithiasis Guidelines" in Turkey and to identify critical factors influencing their application by urologists.

METHODS

The study targeted practicing urologists in Turkey, primarily those involved in the management of urolithiasis, to assess their perspectives and experiences regarding the clinical application of established guidelines. A total of 415 urology specialists were invited to participate in a survey-based study conducted via Google Forms. Participation was voluntary, and 65.08% of the invited urologists completed the survey.

RESULTS

Among the respondents, 84.7% reported utilizing the available guidelines in their routine clinical practice, with varying frequencies of reference. The primary motivations for guideline use were the prevention of potential complications and the avoidance of legal risks, as indicated by 90.5% of respondents. While 56.9% of participants adhered to the guidelines as a clinically standardized practice, 41.6% reported applying the recommendations on a case-by-case basis. Notably, 41.0% of respondents emphasized the need for locally adapted versions of guideline texts. Additionally, nearly half of the participants reported receiving no formal education or training on the significance, content, and practical application of these guidelines. Furthermore, 12.7% expressed skepticism about the evidence-based foundation of the guidelines, questioning whether the recommendations were derived from rigorously conducted studies.

CONCLUSION

The available urolithiasis guidelines are recognized as valuable resources offering key recommendations for the effective and safe management of urolithiasis. However, findings from this survey highlight significant variability in clinical practice patterns due to local conditions and the individual experience and attitudes of practicing urologists. The application of guideline recommendations is further influenced by perceptions regarding their development, content, and practicality. Insights gathered from this study may contribute to improving the preparation, dissemination, and implementation of urolithiasis guidelines, particularly in adapting them to local clinical settings.

Pilot Study of a Modified DOPS Scale for Insulin Pump and CGM Installation Training in Chinese Medical Students During Endocrinology Rotations

Background

Direct Observation of Procedural Skills (DOPS) is a clinical assessment tool that enables trainers to observe medical students' procedural abilities in real-time clinical settings. It assesses students' knowledge application, decision-making, and skill proficiency during clinical tasks.

Methods

This study modifies the DOPS to evaluate the operation of insulin pumps (PUMP) and continuous glucose monitoring systems (CGMS) in diabetes management. Key elements of the modified DOPS include 1) Knowledge Assessment: Evaluating understanding of PUMP and CGMS, including interpreting CGMS data for insulin adjustments; 2) Operational Skills: Assessing correct PUMP needle insertion, programming, and adjustments; 3) Patient Safety: Ensuring safe and aseptic procedures; 4) Feedback: Providing constructive feedback to help students improve their skills.

Results

Training through DOPS led to significant improvements in all domains, overall performance scores, and reduced execution time for each domain. Correlations between domains showed that PUMP indication scores were linked to all other domains and execution times, including re-evaluation. Communication skills and seeking assistance were crucial factors influencing other domains. Multilinear regression analysis revealed that while DOPS-CGMS (R square 1.0) fully explained performance scores, DOPS-PUMP (R square 0.984) indicated that additional personal qualities significantly impacted students' PUMP operation performance.

Conclusion

This customized DOPS form offers insights into students' abilities in managing diabetes with PUMP and CGMS, while emphasizing the need for training on both technical skills and interpersonal skills in future educational models.

"Praise in Public; Criticize in Private": Unwritable Assessment Comments and the Performance Information That Resists Being Written

PURPOSE

Written assessment comments are needed to archive feedback and inform decisions. Regrettably, comments are often impoverished, leaving performance-relevant information undocumented. Research has focused on content and supervisor's ability and motivation to write it but has not sufficiently examined how well the undocumented information lends itself to being written as comments. Because missing information threatens the validity of assessment processes, this study examined the performance information that resists being written.

METHOD

Two sequential data collection methods and multiple elicitation techniques were used to triangulate unwritten assessment comments. Between November 2022 and January 2023, physicians in Canada were recruited by email and social media to describe experiences with wanting to convey assessment information but feeling unable to express it in writing. Fifty supervisors shared examples via survey. From January to May 2023, a subset of 13 participants were then interviewed to further explain what information resisted being written and why it seemed impossible to express in writing and to write comments in response to a video prompt or for their own "unwritable" example. Constructivist grounded theory guided data collection and analysis.

RESULTS

Not all performance-relevant information was equally writable. Information resisted being written as assessment comments when it would require an essay to be expressed in writing, belonged in a conversation and not in writing, or was potentially irrelevant and unverifiable. In particular, disclosing sensitive information discussed in a feedback conversation required extensive recoding to protect the learner and supervisor-learner relationship.

CONCLUSIONS

When documenting performance information as written comments is viewed as an act of disclosure, it becomes clear why supervisors may feel compelled to leave some comments unwritten. Although supervisors can be supported in writing better assessment comments, their failure to write invites a reexamination of expectations for documenting feedback and performance information as written comments on assessment forms.

The Next Era of Assessment Within Medical Education: Exploring Intersections of Context and Implementation

In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.

Integrating Learners Into the Pediatric Primary Care Workflow: Strategies for Optimizing Teaching and Learning

Primary care pediatric providers play an important role in the education of future medical professionals. However, it may feel challenging to integrate a learner into out-patient practice given both time constraints and varying levels of experience among learners. Here we discuss how learners at various stages of training from different medical professions can be integrated into the outpatient pediatric clinical environment. We review eight teaching strategies and provide examples of their use in practice. The goal is to introduce tools to support teachers working with learners in the clinical environment to optimize educational experiences for both teachers and learners. [Pediatr Ann. 2024;53(10):e386-e391.].

Curriculum Innovations: Enhancing Skills in Serious Illness Communication in Neurology Residents Using Simulation: A Pilot Study

Background and Problem Statement

Patients with acute ischemic stroke are faced with prognostic uncertainty, progressive decline, and early mortality. Many neurologists report a lack of education and experience in providing palliative care. We developed a simulation-based curriculum to improve residents' confidence and comfort with conducting late-stage goals of care (GOC) conversations.

Objectives

To assess and improve neurology residents' self-reported confidence and comfort around GOC discussions, prognostication, and hospice; encourage neurology residents to conduct GOC conversations early in the illness; introduce neurology residents to a structured framework for conducting GOC conversations; facilitate the residents to build rapport and convey a mindful presence during GOC conversations; provide direct, real-time feedback and an opportunity for redo and practice; and identify gaps for education.

Methods and Curriculum Description

The 3-hour experience included a didactic session followed by an interactive simulation and debriefing. The residents' objectives were to deliver difficult news, discuss prognosis, explore goals, navigate treatment options, and discuss end-of-life care including hospice. The faculty observed each interaction and called time-outs to allow the residents to self-assess and obtain feedback. Residents and faculty debriefed to identify take-home points and to reflect on their emotions, self-care, and sense of purpose in medicine.

Results and Assessment

Twenty-six neurology residents filled out an anonymous presurvey to self-assess their confidence and comfort surrounding GOC conversations. More than 50% of residents reported being confident in conducting GOC discussions, whereas only 42% reported adequate prior training. Postsession, more than 90% of residents reported that training was relevant, helpful, organized, and clear. Faculty identified that residents had difficulty addressing prognosis, assessing goals, planning treatment, and using silence, responding to emotion, and displaying empathy. Fifteen residents filled out a postsurvey that revealed improved comfort with delivering prognosis, discussing hospice, and initiating early GOC discussions.

Discussion and Lessons Learned

Our project uniquely focuses on late-stage GOC conversations and builds on existing literature that supports a structured program with both didactic and simulation components to improve residents' abilities to effectively navigate GOC conversations with patients and families. Future work will focus on reinforcement and reassessment of communication skills.

Exploring Medical Student Experiences With Direct Observation During the Pediatric Clerkship

PURPOSE

Direct observation (DO) enables assessment of vital competencies, such as clinical skills. Despite national requirement that medical students experience DOs during each clerkship, the frequency, length, quality, and context of these DOs are not well established. This study examines the quality, quantity, and characteristics of DOs obtained during pediatrics clerkships across multiple institutions.

METHOD

This multimethod study was performed at 6 U.S.-based institutions from March to October 2022. In the qualitative phase, focus groups and/or semistructured interviews were conducted with third-year medical students at the conclusion of pediatrics clerkships. In the quantitative phase, the authors administered an internally developed instrument after focus group discussions or interviews. Qualitative data were analyzed using thematic analysis, and quantitative data were analyzed using anonymous survey responses.

RESULTS

Seventy-three medical students participated in 20 focus groups, and 71 (97.3%) completed the survey. The authors identified 7 themes that were organized into key principles: before, during, and after DO. Most students reported their DOs were conducted primarily by residents (62 [87.3%]) rather than attendings (6 [8.4%]) in inpatient settings. Participants reported daily attending observation of clinical reasoning (38 [53.5%]), communication (39 [54.9%]), and presentation skills (58 [81.7%]). One-third reported they were never observed taking a history by an inpatient attending (23 [32.4%]), and one-quarter reported they were never observed performing a physical exam (18 [25.4%]).

CONCLUSIONS

This study revealed that students are not being assessed for performing vital clinical skills in the inpatient setting by attendings as frequently as previously believed. When observers set expectations, create a safe learning environment, and follow up with actionable feedback, medical students perceive the experience as valuable; however, the DO experience is currently suboptimal. Therefore, a high-quality, competency-based clinical education for medical students is necessary to directly drive future patient care by way of a competent physician workforce.

Entrustable Professional Activities in Palliative Medicine: A Faculty and Learner Development Activity

Background: Faculty development (FD) is critical to the implementation of competency-based medical education (CBME) and yet evidence to guide the design of FD activities is limited. Our aim with this study was to describe and evaluate an FD activity as part of CBME implementation. Methods: Palliative medicine faculty were introduced to entrustable professional activities (EPAs) and gained experience estimating a learner's level of readiness for entrustment by directly observing a simulated encounter. The variation that was found among assessments was discussed in facilitated debrief sessions. Attitudes and confidence levels were measured 1 week and 6 months following debriefs. Results: Participants were able to use the EPA framework when estimating the learner's readiness level for entrustment. Significant improvements in attitudes and level of confidence for several knowledge, skill, and behavior domains were maintained over time. Conclusions: Simulated direct observation and facilitated debriefs contributed to preparing both faculty and learners for CBME and EPA implementation.

An Analysis of Workplace-Based Assessments for Core Entrustable Professional Activities for Entering Residency: Does Type of Clinical Assessor Influence Level of Supervision Ratings?

PURPOSE

The authors describe use of the workplace-based assessment (WBA) coactivity scale according to entrustable professional activities (EPAs) and assessor type to examine how diverse assessors rate medical students using WBAs.

METHOD

A WBA data collection system was launched at Oregon Health and Science University to visualize learner competency in various clinical settings to foster EPA assessment. WBA data from January 14 to June 18, 2021, for medical students (all years) were analyzed. The outcome variable was level of supervisor involvement in each EPA, and the independent variable was assessor type.

RESULTS

A total of 7,809 WBAs were included. Most fourth-, third-, and second-year students were assessed by residents or fellows (755 [49.5%], 1,686 [48.5%], and 918 [49.9%], respectively) and first-year students by attending physicians (803 [83.0%]; P < .001). Attendings were least likely to use the highest rating of 4 (1 was available just in case; 2,148 [56.7%] vs 2,368 [67.7%] for residents; P < .001). Learners more commonly sought WBAs from attendings for EPA 2 (prioritize differential diagnosis), EPA 5 (document clinical encounter), EPA 6 (provide oral presentation), EPA 7 (form clinical questions and retrieve evidence-based medicine), and EPA 12 (perform general procedures of a physician). Residents and fellows were more likely to assess students on EPA 3 (recommend and interpret diagnostic and screening tests), EPA 4 (enter and discuss orders and prescriptions), EPA 8 (give and receive patient handover for transitions in care), EPA 9 (collaborate as member of interprofessional team), EPA 10 (recognize and manage patient in need of urgent care), and EPA 11 (obtain informed consent).

CONCLUSIONS

Learners preferentially sought resident versus attending supervisors for different EPA assessments. Future research should investigate why learners seek different assessors more frequently for various EPAs and if assessor type variability in WBA levels holds true across institutions.

Applying a Competency-Based Medical Education Framework to Development of Residents' Feedback Skills

OBJECTIVE

Feedback is a critically important tool in medical education. This pilot program applies and evaluates a competency-based approach to develop residents' skills in providing feedback to medical students.

METHODS

In 2018-2019, a competency-based resident feedback skills program incorporating videorecording of skills, multi-source feedback using assessment tools with validity evidence, and sequential deliberate practice was piloted in a single-center, prospective study at the University of Rochester. Study participants included eight second-year psychiatry residents and 23 third-year clerkship students. After an introduction to foundational feedback concepts in didactic sessions, residents were videorecorded providing feedback to medical students. Recordings were reviewed with a faculty member for feedback. Skills were assessed by students who had received resident feedback, residents, and faculty utilizing a tool with validity evidence. Observations were repeated a total of three times.

RESULTS

Mean feedback scores increased from 2.70 at the first feedback observation, to 2.77 at the second feedback observation, to 2.89 at the third feedback observation (maximum 3.00 points). The differences between the first and third sessions (0.19) and second and third sessions (0.12) were statistically significant (p values were < .001 and .007, with SE of 0.4 and 0.4, respectively).

CONCLUSIONS

The observed competency-based feedback skills training program for residents using sequential, multi-source review and feedback was feasible and effective. Direct observation is a key component of high-quality feedback, and videorecording is an efficient methodology for observations, enabling both direct observation by the assessor and opportunity for enhanced self-assessment by residents viewing themselves in the feedback encounter.

Towards a flexible and personalised development of veterinarians and veterinary nurses working in a companion animal referral care setting

In the Netherlands, the demand for veterinarians and veterinary nurses (VNs) working within referral care is rapidly growing and currently exceeds the amount of available board-certified specialists. Simultaneously, a transparent structure to guide training and development and to assess quality of non-specialist veterinarians and VNs working in a referral setting is lacking. In response, we developed learning pathways guided by an entrustable professional activity (EPA) framework and programmatic assessment to support personalised development and competence of veterinarians and VNs working in referral settings. Between 4 and 35 EPAs varying per discipline (n = 11) were developed. To date, 20 trainees across five disciplines have been entrusted. Trainees from these learning pathways have proceeded to acquire new EPAs in addition to their already entrusted set of EPAs or progressed to specialist training during (n = 3) or after successfully completing (n = 1) the learning pathway. Due to their outcome-based approach, the learning pathways support flexible ways of development.

Fine-Tuning Large Language Models to Enhance Programmatic Assessment in Graduate Medical Education

Background

Natural language processing is a collection of techniques designed to empower computer systems to comprehend and/or produce human language. The purpose of this investigation was to train several large language models (LLMs) to explore the tradeoff between model complexity and performance while classifying narrative feedback on trainees into the Accreditation Council for Graduate Medical Education subcompetencies. We hypothesized that classification accuracy would increase with model complexity.

Methods

The authors fine-tuned several transformer-based LLMs (Bidirectional Encoder Representations from Transformers [BERT]-base, BERT-medium, BERT-small, BERT-mini, BERT-tiny, and SciBERT) to predict Accreditation Council for Graduate Medical Education subcompetencies on a curated dataset of 10 218 feedback comments. Performance was compared with the authors' previous work, which trained a FastText model on the same dataset. Performance metrics included F1 score for global model performance and area under the receiver operating characteristic curve for each competency.

Results

No models were superior to FastText. Only BERT-tiny performed worse than FastText. The smallest model with comparable performance to FastText, BERT-mini, was 94% smaller. Area under the receiver operating characteristic curve for each competency was similar on BERT-mini and FastText with the exceptions of Patient Care 7 (Situational Awareness and Crisis Management) and Systems-Based Practice.

Discussion

Transformer-based LLMs were fine-tuned to understand anesthesiology graduate medical education language. Complex LLMs did not outperform FastText. However, equivalent performance was achieved with a model that was 94% smaller, which may allow model deployment on personal devices to enhance speed and data privacy. This work advances our understanding of best practices when integrating LLMs into graduate medical education.

Are the content and usability of a new direct observation tool adequate for assessing competency in delivering person-centred care: a think-aloud study with patients and healthcare professionals in Sweden

OBJECTIVE

To evaluate the content and usability of a new direct observation tool for assessing competency in delivering person-centred care based on the Gothenburg Centre for Person-Centred Care (gPCC) framework.

DESIGN

This is a qualitative study using think-aloud techniques and retrospective probing interviews and analyzed using deductive content analysis.

SETTING

Sessions were conducted remotely via Zoom with participants in their homes or offices.

PARTICIPANTS

11 participants with lengthy experience of receiving, delivering and/or implementing gPCC were recruited using purposeful sampling and selected to represent a broad variety of stakeholders and potential end-users.

RESULTS

Participants generally considered the content of the four main domains of the tool, that is, person-centred care activities, clinician manner, clinician skills and person-centred care goals, to be comprehensive and relevant for assessing person-centred care in general and gPCC in particular. Some participants pointed to the need to expand person-centred care activities to better reflect the emphasis on eliciting patient resources/capabilities and psychosocial needs in the gPCC framework. Think-aloud analyses revealed some usability issues primarily regarding difficulties or uncertainties in understanding several words and in using the rating scale. Probing interviews indicated that these problems could be mitigated by improving written instructions regarding response options and by replacing some words. Participants generally were satisfied with the layout and structure of the tool, but some suggested enlarging font size and text spacing to improve readability.

CONCLUSION

The tool appears to satisfactorily cover major person-centred care activities outlined in the gPCC framework. The inclusion of content concerning clinician manner and skills was seen as a relevant embellishment of the framework and as contributing to a more comprehensive assessment of clinician performance in the delivery of person-centred care. A revised version addressing observed content and usability issues will be tested for inter-rater and intra-rater reliability and for feasibility of use in healthcare education and quality improvement efforts.

Making goals count: A theory-informed approach to on-shift learning goals

Supervisors often ask emergency medicine trainees for their learning goals at the start of a clinical shift, though they may do so without considering the reasons for this practice. Recognizing the underlying rationale for voicing on-shift learning goals and proactively considering solutions for some of the associated challenges can help learners and supervisors employ this practice to its full potential. Goal articulation is rooted in educational principles such as self-regulated learning, targeted performance feedback, and collaborative relationships between learner and supervisor. Despite the potential for on-shift learning goals to augment learning, there are numerous barriers that make it challenging for learners and supervisors alike to create or follow up on meaningful goals. Learner-related challenges include uncertainty about how to develop goals within an unpredictable clinical environment and creating goals too narrow or broad in scope. Supervisor-related challenges include difficulties integrating direct observation into the clinical workflow and a desire to avoid negative feedback. The learning environment also presents inherent challenges, such as lack of longitudinal supervisor-learner relationships, time constraints, space limitations, and incentives for learners to conceal their knowledge gaps. The authors discuss these challenges to effective on-shift learning goals and propose solutions that target the learner's approach, the supervisor's approach, and the learning environment itself.

Coaching in Competence by Design: A New Model of Coaching in the Moment and Coaching Over Time to Support Large Scale Implementation

Coaching is an increasingly popular means to provide individualized, learner-centered, developmental guidance to trainees in competency based medical education (CBME) curricula. Aligned with CBME's core components, coaching can assist in leveraging the full potential of this educational approach. With its focus on growth and improvement, coaching helps trainees develop clinical acumen and self-regulated learning skills. Developing a shared mental model for coaching in the medical education context is crucial to facilitate integration and subsequent evaluation of success. This paper describes the Royal College of Physicians and Surgeons of Canada's coaching model, one that is theory based, evidence informed, principle driven and iteratively and developed by a multidisciplinary team. The coaching model was specifically designed, fit for purpose to the postgraduate medical education (PGME) context and implemented as part of Competence by Design (CBD), a new competency based PGME program. This coaching model differentiates two coaching roles, which reflect different contexts in which postgraduate trainees learn and develop skills. Both roles are supported by the RX-OCR process: developing Relationship/Rapport, setting eXpectations, Observing, a Coaching conversation, and Recording/Reflecting. The CBD Coaching Model and its associated RX-OCR faculty development tool support the implementation of coaching in CBME. Coaching in the moment and coaching over time offer important mechanisms by which CBD brings value to trainees. For sustained change to occur and for learners and coaches to experience the model's intended benefits, ongoing professional development efforts are needed. Early post implementation reflections and lessons learned are provided.

Education programmes on performance-based assessment for allied health and nursing clinical educators: A scoping review protocol

Background

Performance-based assessment (PBA) is a complex process undertaken in the workplace by healthcare practitioners known as clinical educators, who assist universities in determining health professional students' readiness for independent practice. Preparing healthcare professionals for PBA is considered essential to ensuring the quality of the assessment process in the clinical learning environment. A preliminary search of the literature indicated a paucity of research guiding the development of education programmes that support practice educators to understand and implement PBA.

Objective

The aim of this scoping review is to investigate and describe education programmes delivered to allied health and nursing clinical educators, to develop PBA knowledge and skills.

Methods

This review will follow the Joanna Briggs Institute (JBI) methodology for conducting scoping reviews. Electronic databases relevant to this research topic will be searched including, EMBASE, ERIC, MEDLINE (Ovid), Web of Science and CINAHL and other targeted databases for grey literature. Studies that include PBA as the main focus or a component of the education programmes, of any format, delivered to clinical educators in allied health and nursing will be included. Studies may report the design and/or implementation and/or evaluation of PBA education programmes. Relevant English language publications will be sought from January 2000 to October 2022. Two reviewers will screen all titles and abstracts against the inclusion/exclusion criteria, and publications deemed relevant will be eligible for full text screening, confirming appropriateness for inclusion in the scoping review. Data will be charted to create a table of the results, supported a by narrative summary of the findings in line with the review objectives.

Development of a Tool to Assess Surgical Resident Competence On-Call: The Western University Call Assessment Tool (WUCAT)

BACKGROUND

A central tenet of competency-based medical education is the formative assessment of trainees. There are currently no assessments designed to examine resident competence on-call, despite the on-call period being a significant component of residency, characterized by less direct supervision compared to daytime. The purpose of this study was to design a formative on-call assessment tool and collect valid evidence on its application.

METHODS

Nominal group technique was used to identify critical elements of surgical resident competence on-call to inform tool development. The tool was piloted over six months in the Division of Plastic & Reconstructive Surgery at our institution. Quantitative and qualitative evidence was collected to examine tool validity.

RESULTS

A ten-item tool was developed based on the consensus group results. Sixty-three assessments were completed by seven staff members on ten residents during the pilot. The tool had a reliability coefficient of 0.67 based on a generalizability study and internal item consistency was 0.92. Scores were significantly associated with years of training. We found the tool improved the quantity and structure of feedback given and that the tool was considered feasible and acceptable by both residents and staff members.

CONCLUSIONS

The Western University Call Assessment Tool (WUCAT) has multiple sources of evidence supporting its use in assessing resident competence on-call.

Evaluating Feedback Comments in Entrustable Professional Activities: A Cross-Sectional Study

INTRODUCTION

Competency-based medical education (CBME) has transformed postgraduate medical training, prioritizing competency acquisition over traditional time-based curricula. Integral to CBME are Entrustable Professional Activities (EPAs), that aim to provide high-quality feedback for trainee development. Despite its importance, the quality of feedback within EPAs remains underexplored.

METHODS

We employed a cross-sectional study to explore feedback quality within EPAs, and to examine factors influencing length of written comments and their relationship to quality. We collected and analyzed 1163 written feedback comments using the Quality of Assessment for Learning (QuAL) score. The QuAL aims to evaluate written feedback from low-stakes workplace assessments, based on 3 quality criteria (evidence, suggestion, connection). Afterwards, we performed correlation and regression analyses to examine factors influencing feedback length and quality.

RESULTS

EPAs facilitated high-quality written feedback, with a significant proportion of comments meeting quality criteria. Task-oriented and actionable feedback was prevalent, enhancing value of low-stakes workplace assessments. From the statistical analyses, the type of assessment tool significantly influenced feedback length and quality, implicating that direct and video observations can yield superior feedback in comparison to case-based discussions. However, no correlation between entrustment scores and feedback quality was found, suggesting potential discrepancies between the feedback and the score on the entrustability scale.

CONCLUSION

This study indicates the role of the EPAs to foster high-quality feedback within CBME. It also highlights the multifaceted feedback dynamics, suggesting the influence of factors such as feedback length and assessment tool on feedback quality. Future research should further explore contextual factors for enhancing medical education practices.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

"It's All About the Trust And Building A Foundation:" Evaluation of a Longitudinal Medical Student Coaching Program

Coaching is increasingly implemented in medical education to support learners' growth, learning, and wellbeing. Data demonstrating the impact of longitudinal coaching programs are needed.

We developed and evaluated a comprehensive longitudinal medical student coaching program designed to achieve three aims for students: fostering personal and professional development, advancing physician skills with a growth mindset, and promoting student wellbeing and belonging within an inclusive learning community. We also sought to advance coaches' development as faculty through satisfying education roles with structured training. Students meet with coaches weekly for the first 17 months of medical school for patient care and health systems skills learning, and at least twice yearly throughout the remainder of medical school for individual progress and planning meetings and small-group discussions about professional identity. Using the developmental evaluation framework, we iteratively evaluated the program over the first five years of implementation with multiple quantitative and qualitative measures of students' and coaches' experiences related to the three aims.

The University of California, San Francisco, School of Medicine, developed a longitudinal coaching program in 2016 for medical students alongside reform of the four-year curriculum. The coaching program addressed unmet student needs for a longitudinal, non-evaluative relationship with a coach to support their development, shape their approach to learning, and promote belonging and community.

In surveys and focus groups, students reported high satisfaction with coaching in measures of the three program aims. They appreciated coaches' availability and guidance for the range of academic, personal, career, and other questions they had throughout medical school. Students endorsed the value of a longitudinal relationship and coaches' ability to meet their changing needs over time. Students rated coaches' teaching of foundational clinical skills highly. Students observed coaches learning some clinical skills with them - skills outside a coach's daily practice. Students also raised some concerns about variability among coaches. Attention to wellbeing and belonging to a learning community were program highlights for students. Coaches benefited from relationships with students and other coaches and welcomed the professional development to equip them to support all student needs.

Students perceive that a comprehensive medical student coaching program can achieve aims to promote their development and provide support. Within a non-evaluative longitudinal coach relationship, students build skills in driving their own learning and improvement. Coaches experience a satisfying yet challenging role. Ongoing faculty development within a coach community and funding for the role seem essential for coaches to fulfill their responsibilities.

Effectiveness of a proficiency-based progression e-learning approach to training in communication in the context of clinically deteriorating patients: a multi-arm randomised controlled trial

OBJECTIVE

To determine the effectiveness of proficiency-based progression (PBP) e-learning in training in communication concerning clinically deteriorating patients.

DESIGN

Single-centre multi-arm randomised double-blind controlled trial with three parallel arms.

RANDOMISATION, SETTING AND PARTICIPANTS

A computer-generated program randomised and allocated 120 final year medical students in an Irish University into three trial groups.

INTERVENTION

Each group completed the standard Identification, Situation, Background, Assessment, Recommendation communication e-learning; group 1 Heath Service Executive course group (HSE) performed this alone; group 2 (PBP) performed additional e-learning using PBP scenarios with expert-determined proficiency benchmarks composed of weighted marking schemes of steps, errors and critical errors cut-offs; group 3 (S) (self-directed, no PBP) performed additional e-learning with identical scenarios to (PBP) without PBP.

MAIN OUTCOME MEASURES

Primary analysis was based on 114 students, comparing ability to reach expert-determined predefined proficiency benchmark in standardised low-fidelity simulation assessment, before and after completion of each group's e-learning requirements. Performance was recorded and scored by two independent blinded assessors.

RESULTS

Post-intervention, proficiency in each group in the low-fidelity simulation environment improved with statistically significant difference in proficiency between groups (p<0.001). Proficiency was highest in (PBP) (81.1%, 30/37). Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and self-directed (S) (p<0.001) and (HSE) (p<0.001). No statistically significant difference existed between (S) and (HSE) (p=0.479). Changes in proficiency from pre-intervention to post-intervention were significantly different between the three groups (p=0.001). Post-intervention, an extra 67.6% (25/37) in (PBP) achieved proficiency in the low-fidelity simulation. Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and both (S) (p=0.020) and (HSE) (p<0.001). No statistically significant difference was found between (S) and (HSE) (p=0.156).

CONCLUSIONS

PBP e-learning is a more effective way to train in communication concerning clinically deteriorating patients than standard e-learning or e-learning without PBP.

TRIAL REGISTRATION NUMBER

NCT02937597.

It Takes a Village: Optimal Graduate Medical Education Requires a Deliberately Developmental Organization

Coaching is proposed as a means of improving the learning culture of medicine. By fostering trusting teacher-learner relationships, learners are encouraged to embrace feedback and make the most of failure. This paper posits that a cultural shift is necessary to fully harness the potential of coaching in graduate medical education. We introduce the deliberately developmental organization framework, a conceptual model focusing on three core dimensions: developmental communities, developmental aspirations, and developmental practices. These dimensions broaden the scope of coaching interactions. Implementing this organizational change within graduate medical education might be challenging, yet we argue that embracing deliberately developmental principles can embed coaching into everyday interactions and foster a culture in which discussing failure to maximize learning becomes acceptable. By applying the dimensions of developmental communities, aspirations, and practices, we present a six-principle roadmap towards transforming graduate medical education training programs into deliberately developmental organizations.

Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings

PURPOSE

Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians.

METHOD

Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture.

RESULTS

Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience.

CONCLUSIONS

EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

Examining enablers and barriers to entrustable professional activity acquisition using the theoretical domains framework: A qualitative framework analysis study

Background

Without a clear understanding of the factors contributing to the effective acquisition of high-quality entrustable professional activity (EPA) assessments, trainees, supervising faculty, and training programs may lack appropriate strategies for successful EPA implementation and utilization. The purpose of this study was to identify barriers and facilitators to acquiring high-quality EPA assessments in Canadian emergency medicine (EM) training programs.

Methods

We conducted a qualitative framework analysis study utilizing the Theoretical Domains Framework (TDF). Semistructured interviews of EM resident and faculty participants underwent audio recording, deidentification, and line-by-line coding by two authors, being coded to extract themes and subthemes across the domains of the TDF.

Results

From 14 interviews (eight faculty and six residents) we identified, within the 14 TDF domains, major themes and subthemes for barriers and facilitators to EPA acquisition for both faculty and residents. The two most cited domains (and their frequencies) among residents and faculty were environmental context and resources (56) and behavioral regulation (48). Example strategies to improving EPA acquisition include orienting residents to the competency-based medical education (CBME) paradigm, recalibrating expectations relating to "low ratings" on EPAs, engaging in continuous faculty development to ensure familiarity and fluency with EPAs, and implementing longitudinal coaching programs between residents and faculty to encourage repetitive longitudinal interactions and high-quality specific feedback.

Conclusions

We identified key strategies to support residents, faculty, programs, and institutions in overcoming barriers and improving EPA assessment processes. This is an important step toward ensuring the successful implementation of CBME and the effective operationalization of EPAs within EM training programs.

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

PURPOSE

Faculty feedback on trainees is critical to guiding trainee progress in a competency-based medical education framework. The authors aimed to develop and evaluate a Natural Language Processing (NLP) algorithm that automatically categorizes narrative feedback into corresponding Accreditation Council for Graduate Medical Education Milestone 2.0 subcompetencies.

METHOD

Ten academic anesthesiologists analyzed 5,935 narrative evaluations on anesthesiology trainees at 4 graduate medical education (GME) programs between July 1, 2019, and June 30, 2021. Each sentence (n = 25,714) was labeled with the Milestone 2.0 subcompetency that best captured its content or was labeled as demographic or not useful. Inter-rater agreement was assessed by Fleiss' Kappa. The authors trained an NLP model to predict feedback subcompetencies using data from 3 sites and evaluated its performance at a fourth site. Performance metrics included area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, F1, and calibration curves. The model was implemented at 1 site in a self-assessment exercise.

RESULTS

Fleiss' Kappa for subcompetency agreement was moderate (0.44). Model performance was good for professionalism, interpersonal and communication skills, and practice-based learning and improvement (AUC 0.79, 0.79, and 0.75, respectively). Subcompetencies within medical knowledge and patient care ranged from fair to excellent (AUC 0.66-0.84 and 0.63-0.88, respectively). Performance for systems-based practice was poor (AUC 0.59). Performances for demographic and not useful categories were excellent (AUC 0.87 for both). In approximately 1 minute, the model interpreted several hundred evaluations and produced individual trainee reports with organized feedback to guide a self-assessment exercise. The model was built into a web-based application.

CONCLUSIONS

The authors developed an NLP model that recognized the feedback language of anesthesiologists across multiple GME programs. The model was operationalized in a self-assessment exercise. It is a powerful tool which rapidly organizes large amounts of narrative feedback.

The VOTIS, part 1: development and pilot trial of a tool to assess students' interprofessional skill development using video-reflexive ethnography

This paper explores the development and evaluation of the video Observation Tool for Interprofessional Skills (VOTIS). We describe the development of an authentic interprofessional assessment tool that incorporates video reflection and allows formative and summative assessment of individual learners' interprofessional skills within an authentic interprofessional context. We then investigate its validity and reliability. The VOTIS was developed using a modified Delphi technique. The tool was piloted with 61 students and 11 clinical educators who completed the VOTIS following team meetings where students interacted about their interprofessional clinical work. The following were calculated: internal consistency; students' proficiency levels; inter-rater reliability between students and clinical educators; and inter-rater reliability between clinical educators and an independent rater. Results indicate that the VOTIS has acceptable internal consistency and moderate reliability and has value in evaluating students' interprofessional skills. Study outcomes highlight the need for more explicit wording of tool content and instructions and further clinical educator training to increase the utility and reliability of the VOTIS as a learning and assessment tool.

A phenomenology of direct observation in residency: Is Miller's 'does' level observable?

INTRODUCTION

Guidelines on direct observation (DO) present DO as an assessment of Miller's 'does' level, that is, the learner's ability to function independently in clinical situations. The literature, however, indicates that residents may behave 'inauthentically' when observed. To minimise this 'observer effect', learners are encouraged to 'do what they would normally do' so that they can receive feedback on their actual work behaviour. Recent phenomenological research on patients' experiences with DO challenges this approach; patients needed-and caused-some participation of the observing supervisor. Although guidelines advise supervisors to minimise their presence, we are poorly informed on how some deliberate supervisor participation affects residents' experience in DO situations. Therefore, we investigated what residents essentially experienced in DO situations.

METHODS

We performed an interpretive phenomenological interview study, including six general practice (GP) residents. We collected and analysed our data, using the four phenomenological lenses of lived body, lived space, lived time and lived relationship. We grouped our open codes by interpreting what they revealed about common structures of residents' pre-reflective experiences.

RESULTS

Residents experienced the observing supervisor not just as an observer or assessor. They also experienced them as both a senior colleague and as the patient's familiar GP, which led to many additional interactions. When residents tried to act as if the supervisor was not there, they could feel insecure and handicapped because the supervisor was there, changing the situation.

DISCUSSION

Our results indicate that the 'observer effect' is much more material than was previously understood. Consequently, observing residents' 'authentic' behaviour at Miller's 'does' level, as if the supervisor was not there, seems impossible and a misleading concept: misleading, because it may frustrate residents and cause supervisors to neglect patients' and residents' needs in DO situations. We suggest that one-way DO is better replaced by bi-directional DO in working-and-learning-together sessions.

Optimal training in communication model (OPTiCOM): A programmatic roadmap

OBJECTIVES

Teaching primary care residents patient communication skills is essential, complex, and impeded by barriers. We find no models guiding faculty how to train residents in the workplace that integrate necessary system components, the science of physician-patient communication training and competency-based medical education. The aim of this project is to create such a model.

METHODS

We created OPTiCOM using four steps: (1) communication educator interviews, analysis and theme development; (2) initial model construction; (3) model refinement using expert feedback; (4) structured literature review to validate, refine and finalize the model.

RESULTS

Our model contains ten interdependent building blocks organized into four developmental tiers. The Foundational value tier has one building block Naming relationship as a core value. The Expertize and resources tier includes four building blocks addressing: Curricular expertize, Curricular content, Leadership, and Time. The four building blocks in the Application and development tier are Observation form, Faculty development, Technology, and Formative assessment. The Language and culture tier identifies the final building block, Culture promoting continuous improvement in teaching communication.

CONCLUSIONS

OPTiCOM organizes ten interdependent systems building blocks to maximize and sustain resident learning of communication skills. Practice Implications Residency faculty can use OPTiCOM for self-assessment, program creation and revision.

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment

BACKGROUND

Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique.

OBJECTIVE

To explore faculty's perceptions of the benefits and challenges associated with FORT.

METHODS

Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis.

RESULTS

All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness.

CONCLUSIONS

Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

Evaluating postgraduate family medicine supervisor feedback in registrars' learning portfolios

BACKGROUND

 Postgraduate supervision forms a vital component of decentralised family medicine training. While the components of effective supervisory feedback have been explored in high-income countries, how this construct is delivered in resource-constrained low- to middle-income countries has not been investigated adequately.

AIM

 This article evaluated supervisory feedback in family medicine registrars' learning portfolios (LPs) as captured in their learning plans and mini-Clinical Evaluation Exercise (mini-CEX) forms and whether the training district or the year of training affected the nature of the feedback.

SETTING

 Registrars' LPs from 2020 across five decentralised sites affiliated with the University of the Witwatersrand in South Africa were analysed.

METHODS

 Two modified tools were used to evaluate the quantity of the written feedback in 38 learning plans and 57 mini-CEX forms. Descriptive statistics, Fisher's exact and Wilcoxon rank-sum tests were used for analysis. Content analysis was used to derive counts of areas of feedback.

RESULTS

 Most learning plans (61.2%) did not refer to registrars' clinical knowledge or offer an improvement strategy (86.1%). The 'extent of supervisors' feedback' was rated as 'poor' (63.2%), with only 14.0% rated as 'good.' The 'some' and 'no' feedback categories in the mini-CEX competencies (p  0.001 to p = 0.014) and the 'extent of supervisors' feedback' (p  0.001) were significantly associated with training district. Feedback focused less on clinical reasoning and negotiation skills.

CONCLUSION

 Supervisors should provide specific and constructive narrative feedback and an action plan to improve registrars' future performance.Contribution: Supervisory feedback in postgraduate family medicine training needs overall improvement to develop skilled family physicians.

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students' Self-Assessed Skills by Specialty

BACKGROUND

The Association of American Medical Colleges described 13 Core Entrustable Professional Activities (EPAs) that graduating students should be prepared to perform under indirect supervision on day one of residency. Surgery program directors recently recommended entrustability in these Core EPAs for incoming surgery interns. We sought to determine if graduating students intending to enter surgery agreed they had the skills to perform these Core EPAs.

STUDY DESIGN

Using de-identified, individual-level data collected from and about 2019 Association of American Medical Colleges Graduation Questionnaire respondents, latent profile analysis was used to group respondents based on their self-assessed Core EPAs skills' response patterns. Associations between intended specialty, among other variables, and latent profile analysis group were assessed using independent sample t -tests and chi-square tests and multivariable logistic regression methods.

RESULTS

Among 12,308 Graduation Questionnaire respondents, latent profile analysis identified 2 respondent groups: 7,863 (63.9%) in a high skill acquisition agreement (SAA) group and 4,445 (36.1%) in a moderate SAA group. Specialty was associated with SAA group membership (p < 0.001), with general surgery, orthopaedic surgery, and emergency medicine respondents (among others) overrepresented in the high SAA group. In the multivariable logistic regression models, each of anesthesiology, ophthalmology, pediatrics, psychiatry, and radiology (vs general surgery) specialty intention was associated with a lower odds of high SAA group membership.

CONCLUSION

Graduating students' self-assessed Core EPAs skills were higher for those intending general surgery than for those intending some other specialties. Our findings can inform collaborative efforts to ensure graduates' acquisition of the skills expected of them at the start of residency.

Feedback Delivery in an Academic Cancer Centre: Reflections From an R2C2-based Microlearning Course

Feedback delivery and training have not been characterized in the context of academic cancer centres. The purpose of this study was to assess the feasibility and utility of a microlearning course based on the R2C2 (Relationship, Reaction, Content, Coaching) feedback model and characterize multidisciplinary healthcare provider (HCP) perspectives on existing feedback practices in an academic cancer centre. Five HCP (two radiation oncologists, one medical oncologist, and two allied health professionals) with supervisory roles were selected by purposive sampling to participate in a prospective longitudinal qualitative study. Each participant completed a web-based multimedia course. Semi-structured one-on-one interviews were conducted with each participant at four time points: pre- and immediately post-course, and at one- and three-months post course. All participants found the course to be time feasible and completed it in 10-20 min. Participants expressed that the course fulfilled their need for feedback training and that its adoption may normalize a feedback culture in the cancer centre. Three themes were identified regarding perceptions of existing feedback practices: (1) hierarchical and interdisciplinary relationships modulate feedback delivery, (2) interest in feedback delivery varies by duration of the supervisory relationship, and (3) the transactionality of supervisor-trainee relationships influences feedback delivery. This study demonstrates the perceived feasibility and utility of a digital microlearning approach for development of feedback competencies in an academic cancer centre, perceptions of cultural barriers to feedback delivery, and the need for organizational commitment to developing a feedback culture.

How argumentation theory can inform assessment validity: A critical review

INTRODUCTION

Many health professions education (HPE) scholars frame assessment validity as a form of argumentation in which interpretations and uses of assessment scores must be supported by evidence. However, what are purported to be validity arguments are often merely clusters of evidence without a guiding framework to evaluate, prioritise, or debate their merits. Argumentation theory is a field of study dedicated to understanding the production, analysis, and evaluation of arguments (spoken or written). The aim of this study is to describe argumentation theory, articulating the unique insights it can offer to HPE assessment, and presenting how different argumentation orientations can help reconceptualize the nature of validity in generative ways.

METHODS

The authors followed a five-step critical review process consisting of iterative cycles of focusing, searching, appraising, sampling, and analysing the argumentation theory literature. The authors generated and synthesised a corpus of manuscripts on argumentation orientations deemed to be most applicable to HPE.

RESULTS

We selected two argumentation orientations that we considered particularly constructive for informing HPE assessment validity: New rhetoric and informal logic. In new rhetoric, the goal of argumentation is to persuade, with a focus on an audience's values and standards. Informal logic centres on identifying, structuring, and evaluating arguments in real-world settings, with a variety of normative standards used to evaluate argument validity.

DISCUSSION

Both new rhetoric and informal logic provide philosophical, theoretical, or practical groundings that can advance HPE validity argumentation. New rhetoric's foregrounding of audience aligns with HPE's social imperative to be accountable to specific stakeholders such as the public and learners. Informal logic provides tools for identifying and structuring validity arguments for analysis and evaluation.

Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships

PURPOSE

To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation.

METHOD

This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system.

RESULTS

Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning.

CONCLUSIONS

Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

Feasibility and Effectiveness of Mini-Clinical Evaluation Exercise (Mini-CEX) in an Undergraduate Medical Program: A Study From Pakistan

Background In clinical settings, direct observation (DO) with feedback is an effective method to assess and improve learner performance. One tool used for DO is the mini-clinical evaluation exercise (Mini-CEX). We conducted a study to assess the effectiveness and feasibility of Mini-CEX for medical students at Aga Khan University, Karachi. Methods Utilizing a purposive sampling technique, a total of 199 students in six core clerkships of Years 3 and 4 were selected for this study. Participating faculty underwent training workshops for use of Mini-CEX and feedback strategies. Each student was assessed twice by one faculty, using a modified version of the Mini-CEX, which assessed four domains of clinical skills: Data Gathering, Communication, Diagnosis/Differential, and Management Plan and Organization. Feedback was given after each encounter. Faculty and students also provided detailed feedback regarding the process of DO. Data were analyzed using Statistical Package for Social Sciences (SPSS) version 26 (IBM Corp., Armonk, NY, USA), with categorical variables arranged as frequencies and percentages. The Chi-squared test was used for further statistical analyses, and a P-value of < 0.05 was considered statistically significant. Effectiveness was assessed via a change in student performance between the first and second Mini-CEX, and feasibility was assessed via qualitative feedback. Results We obtained three sets of results: Mini-CEX forms (523), from which we included a total 350 evaluations for analysis, 216 from Year 3 and 134 from Year 4, and feedback on DO: student (70) and faculty (18). Year 3 students performed significantly better in all foci of the Mini-CEX between the first and second assessment (P ≤ 0.001), whereas in Year 4, significant improvement was limited to only two domains of the Mini-CEX [Communication of History/Physical Examination (P = 0.040) and Diagnosis/Differential and Management Plan (P < 0.001)]. Students (65.7%) and faculty (94.4%) felt this exercise improved their interaction. 83.3% faculty recommended its formal implementation compared to 27.1% of students, who reported challenges in implementation of the Mini-CEX such as time constraints, logistics, the subjectivity of assessment, and varying interest by faculty. Conclusion Direct observation using Mini-CEX is effective in improving the clinical and diagnostic skills of medical students and strengthens student-faculty interaction. While challenges exist in its implementation, the strategic placement of Mini-CEX may enhance its utility in measuring student competency.

Does direct observation influence the quality of workplace-based assessment documentation?

Background

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect).

Methods

The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis.

Results

A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84.

Conclusions

Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Interprofessional bedside rounding improves quality of feedback to resident physicians

PURPOSE

Obtaining high quality feedback in residency education is challenging, in part due to limited opportunities for faculty observation of authentic clinical work. This study reviewed the impact of interprofessional bedside rounds ('iPACE™') on the length and quality of faculty narrative evaluations of residents as compared to usual inpatient teaching rounds.

METHODS

Narrative comments from faculty evaluations of Internal Medicine (IM) residents both on usual teaching service as well as the iPACE™ service (spanning 2017-2020) were reviewed and coded using a deductive content analysis approach.

RESULTS

Six hundred ninety-two narrative evaluations by 63 attendings of 103 residents were included. Evaluations of iPACE™ residents were significantly longer than those of residents on usual teams (109 vs. 69 words, p < 0.001). iPACE™ evaluations contained a higher average occurrence of direct observations of patient/family interactions (0.72 vs. 0.32, p < 0.001), references to interprofessionalism (0.17 vs. 0.05, p < 0.001), as well as specific (3.21 vs. 2.26, p < 0.001), actionable (1.01 vs. 0.69, p < 0.001), and corrective feedback (1.2 vs. 0.88, p = 0.001) per evaluation.

CONCLUSIONS

This study suggests that the iPACE™ model, which prioritizes interprofessional bedside rounds, had a positive impact on the quantity and quality of feedback, as measured via narrative comments on weekly evaluations.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool

INTRODUCTION

Clinical reasoning encompasses the process of data collection, synthesis, and interpretation to generate a working diagnosis and make management decisions. Situated cognition theory suggests that knowledge is relative to contextual factors, and clinical reasoning in urgent situations is framed by pressure of consequential, time-sensitive decision-making for diagnosis and management. These unique aspects of urgent clinical care may limit the effectiveness of traditional tools to assess, teach, and remediate clinical reasoning.

METHODS

Using two validated frameworks, a multidisciplinary group of clinicians trained to remediate clinical reasoning and with experience in urgent clinical care encounters designed the novel Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT). REACT is a behaviorally anchored assessment tool scoring five domains used to provide formative feedback to learners evaluating patients during urgent clinical situations. A pilot study was performed to assess fourth-year medical students during simulated urgent clinical scenarios. Learners were scored using REACT by a separate, multidisciplinary group of clinician educators with no additional training in the clinical reasoning process. REACT scores were analyzed for internal consistency across raters and observations.

RESULTS

Overall internal consistency for the 41 patient simulations as measured by Cronbach's alpha was 0.86. A weighted kappa statistic was used to assess the overall score inter-rater reliability. Moderate reliability was observed at 0.56.

DISCUSSION

To our knowledge, REACT is the first tool designed specifically for formative assessment of a learner's clinical reasoning performance during simulated urgent clinical situations. With evidence of reliability and content validity, this tool guides feedback to learners during high-risk urgent clinical scenarios, with the goal of reducing diagnostic and management errors to limit patient harm.

An Online Pattern Recognition-Oriented Workshop to Promote Interest among Undergraduate Students in How Mathematical Principles Could Be Applied within Veterinary Science

Knowing the importance of mathematics and its relationship with veterinary medicine plays an important role for students. To promote interest in this relationship, we developed the workshop “Math in Nature” that utilizes the surrounding environment for stimulating pattern-recognition and observational skills. It consisted of four sections: A talk by a professional researcher, a question-and-answer section, a mathematical pattern identification session, and a discussion of the ideas proposed by students. The effectiveness of the program to raise interest in mathematics was evaluated using a questionnaire applied before and after the workshop. Following the course, a higher number of students agreed with the fact that biological phenomena can be explained and predicted by applying mathematics, and that it is possible to identify mathematical patterns in living beings. However, the students’ perspectives regarding the importance of mathematics in their careers, as well as their interest in deepening their mathematical knowledge, did not change. Arguably, “Math in Nature” could have exerted a positive effect on the students’ interest in mathematics. We thus recommend the application of similar workshops to improve interests and skills in relevant subjects among undergraduate students.

Joining the meta-research movement: A bibliometric case study of the journal Perspectives on Medical Education

PURPOSE

To conduct a bibliometric case study of the journal Perspectives on Medical Education (PME) to provide insights into the journal's inner workings and to "take stock" of where PME is today, where it has been, and where it might go.

METHODS

Data, including bibliographic metadata, reviewer and author details, and downloads, were collected for manuscripts submitted to and published in PME from the journal's Editorial Manager and Web of Science. Gender of authors and reviewers was predicted using Genderize.io. To visualize and analyze collaboration patterns, citation relationships and term co-occurrence social network analyses (SNA) were conducted. VOSviewer was used to visualize the social network maps.

RESULTS

Between 2012-2019 PME received, on average, 260 manuscripts annually (range = 73-402). Submissions were received from authors in 81 countries with the majority in the United States (US), United Kingdom, and the Netherlands. PME published 518 manuscripts with authors based in 31 countries, the majority being in the Netherlands, US, and Canada. PME articles were downloaded 717,613 times (mean per document: 1388). In total 1201 (55% women) unique peer reviewers were invited and 649 (57% women) completed reviews; 1227 (49% women) unique authors published in PME. SNA revealed that PME authors were quite collaborative, with most authoring articles with others and only a minority (n = 57) acting as single authors.

DISCUSSION

This case study provides a glimpse into PME and offers evidence for PME's next steps. In the future, PME is committed to growing the journal thoughtfully; diversifying and educating editorial teams, authors, and reviewers, and liberating and sharing journal data.

Part 5: Essentials of Neonatal-Perinatal Medicine Fellowship: evaluation of competence and proficiency using Milestones

The Accreditation Council for Graduate Medical Education (ACGME) Pediatric Subspecialty Milestone Project competencies are used for Neonatal-Perinatal Medicine (NPM) fellows. Milestones are longitudinal markers that range from novice to expert (levels 1-5). There is no standard approach to the required biannual evaluation of fellows by fellowship programs, resulting in significant variability among programs regarding procedural experience and exposure to pathology during clinical training. In this paper, we discuss the opportunities that Milestones provide, potential strategies to address challenges, and future directions.

Supervisory dyads' communication and alignment regarding the use of workplace-based observations: a qualitative study in general practice residency

BACKGROUND

In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular.

METHODS

We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained.

RESULTS

The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development.

CONCLUSIONS

To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

[Feedback Guide for direct observation of family medicine residents in Canada: a francophone tool]

BACKGROUND

There is no CanMEDS-FM-based milestone tool to guide feedback during direct observation (DO). We have developed a guide to support documentation of feedback for direct observation (DO) in Canadian family medicine (FM) programs.

METHODS

The Guide was designed in three phases with the collaboration of five Canadian FM programs with at least a French-speaking teaching site: 1) literature review and needs assessment; 2) development of the DO Feedback Guide; 3) testing the Guide in a video simulation context with qualitative content analysis.

RESULTS

Phase 1 demonstrated the need for a narrative guide aimed at 1) specifying mutual expectations according to the resident's level of training and the clinical context, 2) providing the supervisor with tools and structure in his observations 3) to facilitate documentation of feedback. Phase 2 made it possible to develop the Guide, in paper and electronic formats, meeting the needs identified. In phase 3, 15 supervisors used the guide for three levels of residence. The Guide was adjusted following this testing to recall the phases of the clinical encounter that were often forgotten during feedback (before consultation, diagnosis and follow-up), and to suggest types of formulation to be favored (stimulating questions, questions of clarification, reflections).

CONCLUSION

Based on evidence and a collaborative approach, this Guide will equip French-speaking Canadian supervisors and residents performing DO in family medicine.