The failure to fail underperforming trainees in health professions education: A BEME systematic review: BEME Guide No. 42

Remediation in clinical practice for student nurses - A scoping review

BACKGROUND

Clinical practice provides an opportunity for undergraduate nursing students to develop the professional attitudes, knowledge and skills required for the delivery of safe competent care. Some students, however, are challenged to consolidate learning in clinical practice and are therefore at risk of failing courses or programmes. Supporting these students requires remediation strategies specific to clinical practice. This is challenging, however, as remediation approaches frequently centre on supporting students in theoretical components of courses/programmes, rather than clinical practice.

OBJECTIVES

A scoping review was conducted to explore and summarise literature in undergraduate nurse education related to remediation support for clinical practice.

METHODS

The research question was developed using the Population/Concept/Context model. Following the identification of keywords, five databases (CINHAL Plus, MEDLINE, ProQuest, Scopus and Informit) were searched. Abstracts and full-text articles were independently screened by two authors. Data from included studies was extracted and then thematically analysed.

RESULTS

Twenty papers met inclusion/exclusion criteria (five literature reviews, one concept analysis, six commentary papers and eight original research studies). Research studies used qualitative, quantitative, or mixed methods research designs. Populations included nurse academics, mentors/preceptors, health professional students (including nursing students) and nursing students exclusively. Three themes were identified: pre-placement remediation strategies; on-placement remediation strategies and post-placement remediation strategies. All authors highlighted the importance of remediation for at-risk students in clinical practice.

CONCLUSIONS

This review identified several remediation strategies that may support undergraduate nursing students in clinical practice. Few, however, were well-defined or rigorously evaluated, highlighting the need for additional research on nursing student remediation in clinical practice. A partnership-based approach to remediation that engages students, educators, and healthcare providers and is underpinned by clear processes may be of further benefit to nursing students in clinical practice.

A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness

INTRODUCTION

Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances.

METHODS

We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity.

RESULTS

Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity.

CONCLUSIONS

The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

'We're like Spider-Man; with great power comes great responsibility': Coaches' experiences supporting struggling medical students

BACKGROUND

Medical students can experience a range of academic and non-academic struggles. Coaching is a valuable strategy to support learners, but coaches describe working with struggling learners as taxing. Transformative learning theory (TLT) provides insights into how educators grow from challenging experiences to build resilience. This study explores how coaches evolve as educators through supporting struggling students.

METHODS

This qualitative study grounded in an interpretivist paradigm used interviews of longitudinal medical student coaches at two academic institutions. Interviews, using TLT as a sensitizing concept, explored coaches' experience coaching struggling learners. We performed thematic analysis.

RESULTS

We interviewed 15 coaches. Coaches described supporting students through multi-faceted struggles which often surprised the coach. Three themes characterized coaches' experiences: personal responsibility, emotional response, and personal learning. Coaches shouldered high personal responsibility for learners' success. For some, this burden felt emotional, raised parental instincts and questions about maintaining boundaries with learners. Coaches evolved their coaching approach, challenged biases, and built skills. Coaches learned to better appreciate the learner point of view and employ resources to support students.

DISCUSSION

Through navigating learner struggles, educators can gain self-efficacy, learn to understand learners' perspectives, and evolve their coaching approach to lessen their personal emotional burden through time.

The missing ingredient: Medical student insights to inform and enhance learner handover

BACKGROUND

Learner handover (LH) is the passing on of information about students between educators. In light of broad acceptance that LH can improve learner support experiences and performance outcomes, those involved are seeking greater governance to achieve practical, effective handover implementation. Stakeholder consultation can inform and enable the co-creation of meaningful, robust practice guidance. This study sought to address the gap in literature around in-depth learner opinion, a key element so far overlooked.

METHODS

This qualitative study (2022) investigated undergraduate medical student perspectives on appropriate tutor information-sharing at the University of Limerick School of Medicine (ULSoM). The findings build upon an educator focus group study published by the authors (2021). Eleven participants were recruited to represent the typical graduate-entry medical school programme population across years 1-4 of study. Their understanding and expectations of "learner handover" were explored qualitatively, using online, individual, semi-structured interviews. Inductive transcript coding and thematic data analysis were applied to illustrate learner insights.

FINDINGS

Emergent themes included shared values, individual context and collaborative process, with ideas proposed for specific action around student education, staff training, mental health support, and documented procedures.

DISCUSSION

Consent, system transparency, data security and the development of positive handover culture were revealed as current needs. Student perspectives, together existing LH literature and highlighted aspects of educational theory, allowed the creation a new conceptual LH framework as a foundation for practice improvement.

CONCLUSION

These findings provide clarity and contextual understanding, mainly from a pre-clinical phase learner standpoint, with pragmatic suggestions to enhance LH appeal.

Marking parties for marking written assessments: A spontaneous community of practice

In programmes of assessment with both high and low-stakes assessments, the inclusion of open-ended long answer questions in the high-stakes examination can contribute to driving deeper learning among students. However, in larger institutions, this would generate a seemingly insurmountable marking workload. In this study, we use a focused ethnographic approach to explore how such a marking endeavour can be tackled efficiently and pragmatically. In marking parties, examiners come together to individually mark student papers. This study focuses on marking parties for two separate tasks assessing written clinical communication in medical school finals at Southampton, UK. Data collected included field notes from 21.3 h of marking parties, details of demographics and clinical and educational experience of examiners, examiners' written answers to an open-ended post-marking party questionnaire, an in-depth interview and details of the actual marks assigned during the marking parties. In a landscape of examiners who are busy clinicians and rarely interact with each other educationally, marking parties represent a spontaneous and sustainable community of practice, with functions extending beyond the mere marking of exams. These include benchmarking, learning, managing biases and exam development. Despite the intensity of the work, marking parties built camaraderie and were considered fun and motivating.

"They were very very nice but just not very good": The interplay between resident-supervisor relationships and assessment in the emergency setting

Purpose

Clinical supervisors hesitate to report learner weaknesses, a widely documented phenomenon referred to as "failure to fail." They also struggle to discuss weaknesses with learners themselves. Their reluctance to report and discuss learner weaknesses threatens the validity of assessment-of-learning decisions and the effectiveness of assessment for learning. Personal and interpersonal factors have been found to act as barriers to reporting learners' difficulties, but the precise role of the resident-supervisor relationship remains underexplored, specifically in the emergency setting. This study aims to better understand if and how factors related to the resident-supervisor relationship are involved in assessment of and for learning in the emergency setting.

Methods

We conducted a qualitative study, using semistructured interviews of 15 clinical supervisors in emergency medicine departments affiliated with our institution. Transcripts were independently coded by three members of the team using an iterative mixed deductive-inductive thematic analysis approach. The team then synthesized the coding and discussed analysis following guidelines for thematic analysis.

Results

Participating emergency medicine supervisors valued resident-supervisor relationships built on collaboration and trust and believed that such relationships support learning. They described how these relationships influenced assessment of and for learning and how in turn assessment influenced the relationship. Almost all profiles of resident-supervisor relationships in our study could hinder the disclosing of resident weaknesses, through a variety of mechanisms. To protect residents and themselves from the discomfort of disclosing weaknesses and to avoid deteriorating the resident-supervisor relationship, many downplayed or even masked residents' difficulties. Supervisors who described themselves as able to provide negative assessment of and for learning often adopted a more distant or professional stance.

Conclusions

This study contributes to a growing literature on failure to fail by confirming the critical impact that the resident-supervisor relationship has on the willingness and ability of emergency medicine supervisors to play their part as assessors.

Dutch dismissal practices: characteristics, consequences, and contrasts in residents' case law in community-based practice versus hospital-based specialties

BACKGROUND

In the Netherlands, 2 to 10% of the residents terminate training prematurely. Infrequently, termination of training is by dismissal. Incidentally, residents may disagree, dispute and challenge these decisions from the programme directors. Resident dismissal is always a difficult decision, most commonly made after, repeated assessments, and triangulation of the resulting assessment data and one or more remediation attempts. Nevertheless, the underlying reasons for dismissal and the policies for remediation and dismissal may differ between training programmes. Such differences may however impact the chance of remediation success, the chance of dismissal and subsequent residents' appeals.

METHOD

We included a total of 70 residents from two groups (community-based and hospital-based specialties) during 10 years of appeals. Subsequently, we compared these groups on factors potentially associated with the outcome of the conciliation board decision regarding the residents' dismissal. We focused herein on remediation strategies applied, and reasons reported to dismiss residents.

RESULTS

In both groups, the most alleged reason to dismiss residents was lack of trainability, > 97%. This was related to deficiencies in professionalism in community-based practice and medical expertise in hospital-based specialties respectively. A reason less frequently mentioned was endangerment of patient care, < 26%. However, none of these residents accused of endangerment, actually jeopardized the patients' health, probably due to the vigilance of their supervisors. Remediation strategies varied between the two groups, whereas hospital-based specialties preferred formal remediation plans in contrast to community-based practice. A multitude of remediation strategies per competency (medical expertise, professionalism, communication, management) were applied and described in these law cases.

DISCUSSION

Residents' appeals in community-based practice were significantly less likely to succeed compared to hospital-based specialties. Hypothesised explanatory factors underlying these differences include community-based practices' more prominent attention to the longitudinal assessment of professionalism, the presence of regular quarterly progress meetings, precise documentation of deficiencies, and discretion over the timing of dismissal in contrast to dismissal in the hospital-based specialties which is only formally possible during scheduled formal summative assessment meetings.

Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

A National Survey Exploring Practices and Perceptions of Sharing Experiential Evaluations With Future Preceptors

OBJECTIVE

This study aimed to understand pharmacy programs' policies and processes of sharing Advanced Pharmacy Practice Experience (APPE) assessment information among preceptors and to determine the types of grading scales currently being used nationally.

METHODS

A 14-question survey was utilized to collect information on Experiential Education policies and practices regarding APPE evaluations and assessment data sharing. The survey was administered electronically to Experiential Education Administrators at accredited schools of pharmacy nationally and gathered information on approach to APPE assessment, information sharing with future preceptors and open-ended responses on how struggling student learners are supported. Descriptive statistics were used to analyze quantitative responses, while qualitative open-ended comments were analyzed using thematic analysis.

RESULTS

A total of 95 responses were included in the analysis (67.9% response rate). The majority of programs (83.2%) reported not sharing student performance assessments with future preceptors. Themes that emerged from the analysis of open-ended comments included concerns about bias and privacy violations, and the benefits of sharing evaluations for preceptor preparation and longitudinal student growth. The grading approach varied, with 53.7% of programs using traditional tiered letter grades and 45.3% using a pass/fail grading system.

CONCLUSION

Most pharmacy programs do not share APPE assessment information with future preceptors due to concerns about bias and protecting student privacy. However, programs may be looking for alternative processes that address the need to facilitate student growth and to support struggling learners. Examples shared may provide stimulus and insight for Experiential Education Offices to engage in programmatic discussions about the approach to assessment sharing practices.

Assessor discomfort and failure to fail in clinical performance assessments

BACKGROUND

Assessment of trainee performance in the workplace is critical to ensuring high standards of clinical care. However, some supervisors find the task to be challenging, and may feel unable to deliver their true judgement on a trainee's performance. They may 'keep MUM' (that is, keep mum about undesirable messages) and fail to fail an underperforming trainee. In this study, we explore the effect of discomfort on assessors.

METHODS

Using a survey method, supervisors of trainees in the Australasian College of Dermatologists were asked to self-report experiences of discomfort in various aspects of trainee workplace assessment and for their engagement in MUM behaviours including failure to fail.

RESULTS

Sixty-one responses were received from 135 eligible assessors. 12.5% of assessors self-reported they had failed to fail a trainee and 18% admitted they had grade inflated a trainee's score on a clinical performance assessment in the previous 12-month period. Assessors who reported higher levels of discomfort in the clinical performance assessment context were significantly more likely to report previously failing to fail a trainee. The study did not reveal significant associations with assessor demographics and self-reports of discomfort or MUM behaviours.

CONCLUSIONS

This study reveals the impact of assessor discomfort on the accuracy of assessment information and feedback to trainees, including as a contributing factor to the failure to fail phenomenon. Addressing assessor experience of discomfort offers one opportunity to impact on the complex and multifactorial issue that failure to fail represents.

"We just don't have the resources": Supervisor perspectives on introducing workplace-based assessments into medical specialist training in South Africa

BACKGROUND

South Africa (SA) is on the brink of implementing workplace-based assessments (WBA) in all medical specialist training programmes in the country. Despite the fact that competency-based medical education (CBME) has been in place for about two decades, WBA offers new and interesting challenges. The literature indicates that WBA has resource, regulatory, educational and social complexities. Implementing WBA would therefore require a careful approach to this complex challenge. To date, insufficient exploration of WBA practices, experiences, perceptions, and aspirations in healthcare have been undertaken in South Africa or Africa. The aim of this study was to identify factors that could impact WBA implementation from the perspectives of medical specialist educators. The outcomes being reported are themes derived from reported potential barriers and enablers to WBA implementation in the SA context.

METHODS

This paper reports on the qualitative data generated from a mixed methods study that employed a parallel convergent design, utilising a self-administered online questionnaire to collect data from participants. Data was analysed thematically and inductively.

RESULTS

The themes that emerged were: Structural readiness for WBA; staff capacity to implement WBA; quality assurance; and the social dynamics of WBA.

CONCLUSIONS

Participants demonstrated impressive levels of insight into their respective working environments, producing an extensive list of barriers and enablers. Despite significant structural and social barriers, this cohort perceives the impending implementation of WBA to be a positive development in registrar training in South Africa. We make recommendations for future research, and to the medical specialist educational leaders in SA.

An Analysis of Written and Numeric Scores in End-of-Rotation Forms from Three Residency Programs

Introduction

End-of-Rotation Forms (EORFs) assess resident progress in graduate medical education and are a major component of Clinical Competency Committee (CCC) discussion. Single-institution studies suggest EORFs can detect deficiencies, but both grades and comments skew positive. In this study, we sought to determine whether the EORFs from three programs, including multiple specialties and institutions, produced useful information for residents, program directors, and CCCs.

Methods

Evaluations from three programs were included (Program 1, Institution A, Internal Medicine: n = 38; Program 2, Institution A, Anesthesia: n = 9; Program 3, Institution B, Anesthesia: n = 11). Two independent researchers coded each written comment for relevance (specificity and actionability) and orientation (praise or critical) using a standardized rubric. Numeric scores were analyzed using descriptive statistics.

Results

4869 evaluations were collected from the programs. Of the 77,434 discrete numeric scores, 691 (0.89%) were considered "below expected level." 71.2% (2683/3767) of the total written comments were scored as irrelevant, while 3217 (85.4%) of total comments were scored positive and 550 (14.6%) were critical. When combined, 63.2% (n = 2379) of comments were scored positive and irrelevant while 6.5% (n = 246) were scored critical and relevant.

Discussion

<1% of comments indicated below average performance; >70% of comments scored irrelevant. Critical, relevant comments were least frequently observed, consistent across all 3 programs. The low rate of constructive feedback and the high rate of irrelevant comments are inadequate for a CCC to make informed decisions. The consistency of these findings across programs, specialties, and institutions suggests both local and systemic changes should be considered.

Longitudinal assessment of undergraduate dental students: Building evidence for validity

PURPOSE

To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations.

METHODS

Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha.

RESULTS

Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included.

CONCLUSIONS

There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.

Twelve tips for improving the quality of assessor judgements in senior medical student clinical assessments

Assessment of senior medical students is usually calibrated at the level of achieving expected learning outcomes for graduation. Recent research reveals that clinical assessors often balance two slightly different perspectives on this benchmark. The first is the formal learning outcomes at graduation, ideally as part of a systematic, program-wide assessment approach that measures learning achievement, while the second is consideration of the candidate's contribution to safe care and readiness for practice as a junior doctor. The second is more intuitive to the workplace, based on experience working with junior doctors. This perspective may enhance authenticity in assessment decisions made in OSCEs and work-based assessments to better align judgements and feedback with professional expectations that will guide senior medical students and junior doctors' future career development. Modern assessment practices should include consideration of qualitative as well as quantitative information, overtly including perspectives of patients, employers, and regulators. This article presents 12 tips for how medical education faculty might support clinical assessors by capturing workplace expectations of first year medical graduates and develop graduate assessments based on a shared heuristic of 'work-readiness'. Peer-to-peer assessor interaction should be facilitated to achieve correct calibration that 'merges' the differing perspectives to produce a shared construct of an acceptable candidate.

Establishing the predictive validity of the intercollegiate membership of the Royal Colleges of surgeons written examination: MRCS part B

The Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) is a high-stakes postgraduate examination taken by thousands of surgical trainees worldwide every year. The MRCS is a challenging assessment, highly regarded by surgical training programmes and valued as a gatekeeper to the surgical profession. The examination is taken at considerable personal, social and financial cost to surgical trainees, and failure has significant implications for career progression. Given the value placed on MRCS, it must be a reliable and valid assessment of the knowledge and skills of early-career surgeons. Our first article 'Establishing the Predictive Validity of the Intercollegiate Membership of the Royal Colleges of Surgeons Written Examination: MRCS Part A' discussed the principles of assessment reliability and validity and outlined the mounting evidence supporting the predictive validity of the MRCS Part A (the multiple-choice questionnaire component of the examination). This, the second article in the series discusses six recently published studies investigating the predictive validity of the MRCS Part B (the clinical component of the examination). All national longitudinal cohort studies reviewed have demonstrated significant correlations between MRCS Part B and other assessments taken during the UK surgical training pathway, supporting the predictive validity of MRCS Part B. This review will be of interest to trainees, trainers and Royal Colleges given the value placed on the examination by surgical training programmes.

Assessing the utility of a novel entrustment-supervision assessment tool

BACKGROUND

Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools.

METHODS

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor.

RESULTS

A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8.

CONCLUSION

The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

Nurse mentors' experiences with suitability assessments in clinical placement: A qualitative study

AIM

This study aims to explore mentors' experiences with assessing the suitability of nursing students in clinical placements in a context where suitability assessment is required by law.

BACKGROUND

It is the schools' responsibility to assess students, but mentors play an important role when students are in clinical placements. Clinical placements are typical arenas for assessing suitability.

DESIGN

An explorative qualitative study METHODS: Participants were 17 registered nurses with experience as mentors for nursing students in clinical placement. The participants represented somatic and surgical wards from three different hospitals in Norway. Individual, semi-structured interviews were conducted.

RESULTS

The analysis revealed a total of four themes: 1) the understanding of suitability, 2) experiencing borderline performance, 3) the heavy responsibility of doing the assessment and 4) the importance of support.

CONCLUSIONS

The mentors found it difficult to assess suitability. They reported having little or no knowledge about leading rules and guidelines. The mentors experienced support from their colleagues, something which lightened the burden of assessing the students. This study indicates that mentors need more information about suitability and suitability assessments.

REGISTRATION NUMBER

To be included in abstract after acceptance.

Perception and engagement in unprofessional behaviors of medical students and residents: a mixed-method study

BACKGROUND

The present study aimed to investigate perception and engagement in unprofessional behavior of residents and medical interns and explore the factors affecting their engagement in unprofessional behavior.

METHOD

This study has an explanatory (quantitative-qualitative) mixed-method design. This study was conducted at Shahid Sadoughi University of Medical Sciences in 2022-2023. Participants, including residents and medical interns (n = 169), were entered by stratified random sampling. A survey was conducted in the quantitative step. A by an unprofessional behavior in clinical practice questionnaire (29 items) was used. For each behavior, the participants were asked to report whether they (a) participated in the behavior and (b) stated that the behavior Is unprofessional. In the qualitative step, 17 participants contributed. The qualitative data were collected by semi-structured interviews and analyzed according to the conventional content analysis approach Graneheim and Lundman introduced.

RESULTS

The highest ratio of participants' engagement in unprofessional behavior was reported in 'failure to introduce yourself and nurses and physician assistants to the patient and his family' (n = 145 (85.8%)). The results showed the proportion of participants who engaged in unprofessional behavior more than those who did not participate. There were associations between participants' engagement in each behavior and their perception of that particular behavior as unprofessional. (p = 0.0001). In the following behaviors, although the participants acknowledged that these behaviors were unprofessional, those who participated in the unprofessional behaviors were significantly more than those who did not participate: failure to comply with clinic regulations and policy (p = 0.01), eating or drinking in the hallway of the clinic (p = 0.01), medical negligence in duties in the clinic setting (p = 0.04) and failure to perform duties in teamwork (p = 0.04). The qualitative results were explored in a theme entitled "internalized unprofessional culture," including three categories "encouraging contextual risk factors towards unprofessionalism," "suppressing of unprofessionalism reporting," and "disbelieving professionalism as a key responsibility."

CONCLUSION

The results indicated that most participants engaged in unprofessional behaviors. The findings resulted from the internalized unprofessional culture in the workplace. The findings showed that engagement in unprofessional behaviors resulted from personal and systemic factors. The weakness of responsibility recognition and identity formation as a professional facilitated the engagement in unprofessional behaviors at the personal level. Furthermore, systemic factors including the contextual risk factors (such as deficiency of explicit and hidden curriculum), and the suppression of unprofessionalism reporting mechanism as a hidden factor played an important role in normalizing unprofessional behavior and promoting engagement in unprofessional behaviors among the participants. Recognition of the nature and extent of students' unprofessional behaviors facilitates educational discussion among teachers and students in this field. The results might assist to establish an assessment system and feedback mechanism to solve the problem of the "failure to fail" problem. In addition, these results provide medical educators insights into the development of professional courses that equip learners with adherence to professionalism and coping skills to deal with unprofessionalism in the healthcare system.

Comparison Between Surgical Trainee Self-Assessment With Supervisor Scores in Technical and Nontechnical Skills to Identify Underperforming Trainees

OBJECTIVE

This study aims to compare trainee self-assessment with supervisor assessment to identify differences in correlations of the demographic data and evaluate whether the instrument can be utilized to identify underperforming trainees.

DESIGN

A novel instrument was designed based off the Royal Australasian College of Surgeons original 9 competencies utilizing the JDocs framework and covers 48-items across all surgical competencies. A multiple regression model using age, gender, postgraduate year, IMG status, and level of training as the variables was performed with backwards elimination, and pairwise comparisons made to identify the degree and direction of influence each variable contributed to trainee and supervisor ratings.

SETTING

Surgical trainees employed in tertiary centers within the Australian Capital Territory and South-East New South Wales health network in Australia.

PARTICIPANTS

A total of 36 of 59 (61%) trainees responded. Two grossly incomplete responses were excluded from further analysis resulting in 34 completed self-assessments for analysis. There was a total of 68 supervisor assessments provided by 25 different nominated supervisors. Of these assessments, 67 were fully completed and one was partially complete.

RESULTS

We identified that for both self-ratings and supervisor ratings, the most significant correlation is with the postgraduate year of the trainee, with correlation identified in 7 out of 9 competencies, although in different competency domains. International Medical Graduate status is associated in 2 of 9 self-ratings, and in 3 of 9 supervisor ratings. Underperforming trainees were able to be identified through supervisor assessment but not self-assessment.

CONCLUSION

The supervisor assessment form was able to identify underperforming trainees. Our findings resonate with existing literature in other specialty fields, and surgical units that employ assessment forms should feel more confident in the interpretation of the data and provision of feedback to trainees.

Demise of the USMLE Step-2 CS exam: Rationalizing a way forward

The COVID-19 pandemic has compelled rethinking and changes in medical education, the most controversial perhaps being the cancelation of USMLE Step-2 Clinical Skills exam (Step-2 CS). What started in March of 2020 as suspension of this professional licensure exam, because of concerns about infection risk for examinees, standardized patients (SPs), and administrators, soon became permanent cancelation in January 2021. Expectedly, it triggered debate in medical education circles. Positively, however, the USMLE regulatory agencies (NBME and FSMB) saw an opportunity to innovate an exam tainted with perceptions of validity deficits, cost, examinee inconvenience, and worries about future pandemics; they therefore called for a public debate to fashion a way forward. We have approached the issue by defining Clinical Skills (CS), exploring its epistemology and historic evolution, including assessment modalities from Hippocratic times to the modern era. We defined CS as the art of medicine manifest in the physician-patient encounter as history taking (driven by communication skills and cultural competence) and physical examination. We classified CS components into knowledge and psychomotor skill domains, established their relative importance in the physician process (clinical reasoning) of diagnosis, thus establishing a theoretical framework for developing valid, reliable, feasible, fair, and verifiable CS assessment. Given the concerns for COVID-19 and future pandemics, we established that CS can largely be assessed remotely, and what could not, can be assessed locally (school/regional consortia level) as part of a USMLE-regulated/supervised assessment regimen with established national standards, thus maintaining USMLE's fiduciary responsibilities. We have suggested a national/regional program for faculty development in CS curriculum development, and assessment, including standard setting skills. This pool of expert faculty will form the nucleus of our proposed USMLE-regulated External Peer Review Initiative (EPRI). Finally, we suggest that CS evolves into an academic discipline/department of its own, rooted in scholarship.

Identifying self-presentation components among nursing students with unsafe clinical practice: a qualitative study

BACKGROUND

Maintaining patient safety is a practical standard that is a priority in nursing education. One of the main roles of clinical instructors is to evaluate students and identify if students exhibit unsafe clinical practice early to support their remediation. This study was conducted to identify self-presentation components among nursing students with unsafe clinical practice.

METHODS

This qualitative study was conducted with 18 faculty members, nursing students, and supervisors of medical centers. Data collection was done through purposive sampling and semi-structured interviews. Data analysis was done using conventional qualitative content analysis using MAXQDA10 software.

RESULTS

One main category labelled self-presentation emerged from the data along with three subcategories of defensive/protective behaviors, assertive behaviors, and aggressive behaviors.

CONCLUSION

In various clinical situations, students use defensive, assertive, and aggressive tactics to maintain their professional identity and present a positive image of themselves when they make a mistake or predict that they will be evaluated on their performance. Therefore, it seems that the first vital step to preventing unsafe behaviors and reporting medical errors is to create appropriate structures for identification, learning, guidance, and evaluation based on progress and fostering a growth mindset among students and clinical educators.

Assessing trainee's performance using reported observations: Perceptions of nurse meta-assessors

BACKGROUND

Educational and health care organizations who prepare meta-assessors to fulfill their role in the assessment of trainees' performance based on reported observations have little literature to rely on. While the assessment of trainees' performance based on reported observations has been operationalized, we have yet to understand the elements that can affect its quality fully. Closing this gap in the literature will provide valuable insight that could inform the implementation and quality monitoring of the assessment of trainees' performance based on reported observations.

OBJECTIVES

The purpose of this study was to explore the elements to consider in the assessment of trainees' performance based on reported observations from the perspectives of meta-assessors.

METHODS

Design, Settings, Participants, data collection and analysis. The authors adopted Sandelowski's qualitative descriptive approach to interview nurse meta-assessors from two nursing programs. A semi-structured interview guide was used to document the elements to consider in the assessment of nursing trainees' performance based on reported observations, and a survey was used to collect sociodemographic data. The authors conducted a thematic analysis of the interview transcripts.

RESULTS

Thirteen meta-assessors participated in the study. Three core themes were identified: (1) meta-assessors' appropriation of their perceived assessment roles and activities, (2) team climate of information sharing, and (3) challenges associated with the assessment of trainees' performance based on reported observations. Each theme is comprised of several sub themes.

CONCLUSIONS

To optimize the quality of the assessment of the trainee's performance based on reported observations and ratings, HPE programs might consider how to clarify better the meta-assessor's roles and activities, as well as how interventions could be created to promote a climate of information sharing and to address the challenges identified. This work will guide educational and health care organizations for better preparation and support for meta-assessors and preceptors.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

After the Discontinuation of Step 2 CS: A Collaborative Statement from the Directors of Clinical Skills Education (DOCS)

Issue: The United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), the only clinical skills competency testing required for licensure in the United States, has been discontinued. Evidence: This exam, though controversial, propelled a movement emphasizing the value of clinical skills instruction and assessment in undergraduate medical education. While disappointed by the loss of this national driver that facilitated standardization of clinical skills education, the Directors of Clinical Skills Education (DOCS) see prospects for educational innovation and growth. DOCS is a national organization and inclusive community of clinical skills education leaders. This statement from DOCS regarding the discontinuation of USMLE Step 2 CS has been informed by DOCS meetings, listserv discussions, an internal survey, and a review of recent literature. Implications: Rigorous clinical skills assessment remains central to effective and patient-centered healthcare. DOCS shares specific concerns as well as potential solutions. Now free from the external pressure to prepare students for success on Step 2 CS, clinical skills educators can reprioritize content and restructure clinical skills programs to best meet the needs of learners and the ever-evolving healthcare landscape. DOCS, as an organization of clinical skills leaders, makes the following recommendations: 1) Collaboration amongst institutions must be prioritized; clinical skills assessment consortia should be expanded. 2) Governing, accrediting, and licensing organizations should leverage their influence to support and require high quality clinical skills assessments. 3) UME clinical skills leaders should develop ways to identify students who perform with exceptional, borderline, and poor clinical skills at their local institutions. 4) UME leadership should fully commit resources and curricular time to graduate students with excellent clinical skills.

Shifting the narrative and practice of assessing professionalism in dietetics education: An Australasian qualitative study

AIM

We aimed to explore current approaches to assessing professionalism in dietetics education in Australia and New Zealand, and asked the questions what is working well and what needs to improve?

METHOD

We employed a qualitative interpretive approach and conducted interviews with academic and practitioner (workplace-based) educators (total sample n = 78) with a key stake in dietetics education across Australia and New Zealand. Data were analysed using team-based, framework analysis.

RESULTS

Our findings suggest significant shifts in dietetics education in the area of professionalism assessment. Professionalism assessment is embedded in formal curricula of dietetics programs and is occurring in university and placement settings. In particular, advances have been demonstrated in those programs assessing professionalism as part of the programmatic assessment. Progress has been enabled by philosophical and curricula shifts; clearer articulation and shared understandings of professionalism standards; enhanced learner agency and reduced power distance; early identification and intervention of professionalism lapses; and increased confidence and capabilities of educators.

CONCLUSIONS

These findings suggest there have been considerable advances in professionalism assessment in recent years with shifts in practice in approaching professionalism through a more interpretivist lens, holistically and more student-centred. Professionalism assessment in dietetics education is a shared responsibility and requires further development and transformation to more fully embed and strengthen curricula approaches across programs. Further work should investigate strategies to build safer learning cultures and capacity for professionalism conversations and in strengthening approaches to remediation.

Failure to fail - Perspective from junior faculty preceptors on the challenges of evaluating underperforming students in the experiential learning environment

INTRODUCTION

Preceptors serve as the gatekeepers between individuals progressing from student pharmacist to independent practitioner. This responsibility is challenging if a student is not progressing as required and is at risk of failure. In this article, we will review the potential consequences and barriers of "failing to fail" a student, share the emotions that come with the decision, and suggest actions that may aid in preceptor decision-making.

COMMENTARY

A preceptor's failure to fail a student has a global impact on many parties including the student, their future employer and patients, the preceptor, and the school or college of pharmacy. Despite supportive factors, preceptors may experience an internal struggle about the ripple effect of failing or not failing an experiential student.

IMPLICATIONS

Underperformance in the experiential setting is a complex issue that remains largely unseen due to "failure to fail," a concept that should be researched further in the pharmacy setting. Empowering preceptors, particularly newer preceptors, to assess and manage failing students is possible through increased discussion around the topic and focused preceptor development programs.

Necessary but Insufficient and Possibly Counterproductive: The Complex Problem of Teaching Evaluations

The evaluation of clinical teachers' performance has long been a subject of research and debate, yet teaching evaluations (TEs) by students remain problematic. Despite their intuitive appeal, there is little evidence that TEs are associated with students' learning in the classroom or clinical setting. TEs are also subject to many forms of bias and are confounded by construct-irrelevant factors, such as the teacher's physical attractiveness or personality. Yet they are used almost exclusively as evaluations of and feedback to teachers. In this commentary, the authors review the literature on what TEs are meant to do, what they actually do in the real world, and their overall impact. The authors also consider productive ways forward. While TEs are certainly necessary to provide the crucial student voice, they are insufficient as the sole way to assess teachers. Further, they are often counterproductive. TEs carry so much weight for faculty that they can act as a disincentive for teachers to challenge learners and provide them with the critical feedback they often need, lest students give them poor ratings. To address these challenges, changes are needed, including embedding TEs in a programmatic assessment framework. For example, TEs might be used for formative feedback only, while other sources of data, such as peer assessments, learning outcomes, 360-degree feedback, and teacher reflections, could be collated into a portfolio to provide a more meaningful evaluation for teachers. Robust, transparent systems should be in place that dictate how TE data are used and to ensure they are not misused. Clinical teachers who do not "fail to fail" learners but instead take the time and effort to identify and support learners in difficulty should be recognized and rewarded. Learners need this support to succeed and the obligation to protect patients demands it.

Reliability and validity testing of the medicines related - consultation assessment tool for assessing pharmacists' consultations

BACKGROUND

Demonstrating a person-centred approach in a consultation is a key component of delivering high-quality healthcare. To support development of such an approach requires training underpinned by valid assessment tools. Given the lack of a suitable pharmacy-specific tool, a new global consultation skills assessment tool: the medicines related-consultation assessment tool (MR-CAT) was designed and tested.

AIM

This study aimed to test the validity and reliability of the MR-CAT using psychometric methods.

METHOD

Psychometric testing involved analysis of participants' (n = 13) assessment of fifteen pre-recorded simulated consultations using the MR-CAT. Analysis included discriminant validity testing, intrarater and interrater reliability testing for each of the five sections of the MR-CAT and for the overall global assessment of the consultation. Analysis also included internal consistency testing for the whole tool.

RESULTS

Internal consistency for the overall global assessment of the consultation was good (Cronbach's alpha = 0.97). The MR-CAT discriminated well for the overall global assessment of the consultation (p < 0.001). Moderate to high intrarater reliability was observed for the overall global assessment of the consultation and for all five sections of the MR-CAT (rho = 0.64-0.84) in the test-retest analysis. Moderate to good interrater reliability (Kendall's W = 0.68-0.90) was observed for the overall global assessment of the consultation and for all five sections of the MR-CAT.

CONCLUSION

The MR-CAT is a valid and reliable tool for assessing person-centred pharmacist's consultations. Moreover, its unique design means that the MR-CAT can be used in both formative and summative assessment.

Supporting health professional educators in the workplace: A scoping review

PURPOSE

Clinical educators frequently request additional support for educating pre-qualification health professions students despite having access to professional development programs to build education knowledge and skills. The breadth of 'additional support' options remains unclear. The aim of this review is to explore what is known about support options for health professional educators in the workplace through the lens of learning organisations.

MATERIALS AND METHODS

A scoping review was conducted searching Ovid Medline, CINAHL, ProQuest and PsycINFO electronic databases from 1 January 2005 up to 21 October 2020 for studies that identified support strategies for clinical educators of pre-qualification students in the workplace. Relevant data were charted, summarised thematically and synthesised with reference to support type and implementation level.

RESULTS

Fifty relevant records related to medicine, nursing and allied health clinical education were included. Twelve support themes and five cross-cutting support categories were identified across four implementation levels of healthcare systems.

CONCLUSIONS

A diversity of support for clinical educators beyond professional development was identified. Future research combined with leadership and commitment from the healthcare and education sectors is needed to better understand the applicability, efficacy and resourcing of any newly integrated support to ensure it is sustainable and improves clinical educator capability.

"Dismantling Fixed Time, Variable Outcome Education: Abandoning 'Ready or Not, Here they Come' is Overdue"

Two decades after competency-based medical education appeared in the lexicon of medical educators, the community continues to struggle with realizing its full potential. The implementation of the time variable, fixed outcome component has languished based on complexity compounded by resistance to change. Learners continue to transition from medical school to residency, and then practice, primarily based on time rather than having achieved the ability to meet the needs of the patient populations they will serve. Only those few who demonstrate glaring deficiencies do not graduate. The authors urge the medical education community to move from the current fixed time path of medical education toward the implementation of a true continuum of time variable, fixed outcome education, training, and deliberate practice. The latter is defined by purposeful learning, coaching, feedback, and repetition on the path to achieving and maintaining expertise. The opportunities afforded by such a time-variable, fixed outcome approach include: 1) development of a career long growth mindset, 2) ability to address evolving population health needs and careers within the context of one's practice, and 3) continual improvement of care quality and outcomes for patients on the journey towards expertise for providers.

Failing to fail phenomena

INTRODUCTION

Clinical competence is the backbone of competence-based dental education. Over time, there has been a paradigm shift toward training students who are capable of independent practice, as opposed to mere academic success.

METHODS

A mixed-method study was undertaken by anonymised email questionnaire to all restorative tutors at the UK Dental School. Demographics and teaching experience were ascertained, along with key questions on the utilisation of online assessment software iDentity. The assessment process for tutors was explored, and barriers experienced when grading students were reported.

RESULTS

The questionnaire was sent to all 51 restorative tutors with a response rate of 59% (n = 30). Only 3.5% of tutors provided verbal feedback and grading to students in person, with 20.7% only completing iDentity gradings following an email reminder. The majority of staff (93.3%) felt comfortable in raising concerns; however, one of the three clinical tutors admitted they had allowed a failing student to a pass. Qualitative analysis demonstrated several themes why tutors were reluctant to fail students: maintaining good relationships, limited supervision, time delay of grading, one-off event and the student's first attempt.

CONCLUSIONS

Grading students as competent as a one-off experience could potentially mask a recurring problem with a student, in turn impacting the student's ability to assess their own weakness and believe themselves to be competent, and potentially be overconfident. Fair and accurate assessment has a significant benefit to student and staff, enabling targeted development to motivate the students and improve the quality of care provided to the patients.

Peer overmarking and insufficient diagnosticity: the impact of the rating method for peer assessment

The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page essay on an air pollutant. A first cohort (N = 260) relied on analytical rating using criteria to assess their peers' essays. A total of 1297 evaluations were made, and each essay received at least four peer ratings. Results indicate a small correlation between peer and teacher marks, and three essays of substandard quality were not recognised by the group of peer raters. A second cohort (N = 230) used comparative judgement. They completed 1289 comparisons, from which a rank order was calculated. Results suggest a large correlation between the university teacher marks and the peer scores and acceptable reliability of the rank order. In addition, the three essays of substandard quality were discerned as such by the group of peer raters. Although replication research is warranted, the results provide the first evidence that, when peer raters overmark and fail to identify substandard work using analytical rating with criteria, university teachers may consider changing the rating method of the peer assessment to comparative judgement.

Learning, judgement and workforce tensions impact failure to fail in a medical postgraduate setting: A supervisor perspective

PURPOSE

Medical training organisations have a duty to prepare medical graduates for future safe, competent practice. Decisions about underperformance are high stakes at the postgraduate level and failure to fail can occur. We aimed to explore this concept from a systems and supervisor perspective.

METHOD

Supervisors of specialist physician trainees were invited to provide written feedback on failure to fail as part of a broader anonymous supervisor survey. They were provided with a trigger statement and responded in free-text format. A deductive content analysis was undertaken through the lenses of supervisor and institution.

RESULTS

Of 663 supervisors who responded to the broader survey, 373 (56%) provided feedback on the failure to fail trigger statement. Analyses indicated an interplay between trainee and supervisor characteristics, and broader system elements. System elements that contributed to failure to fail trainees included lack of longitudinal monitoring and quality of assessment information. Supervisor characteristics included confident, conflicted and avoidant behaviours towards underperforming trainees.

CONCLUSIONS

Individual and system challenges that contributed to failure to fail were identified in this study, and we propose a three-way tension among learning, judgement and workforce. Three potential mitigation strategies have been identified to reduce failure to fail, namely a stage-based approach to remediation, faculty development in supervisory skills and improved assessment-for-learning processes.

Cancel culture: exploring the unintended consequences of cancelling the Canadian national licensing clinical examination

Assessment drives learning. However, when it comes to high-stakes examinations (e.g., for licensure or certification), these assessments of learning may be seen as unnecessary hurdles by some. Licensing clinical skills assessment in particular have come under fire over the years. Recently, assessments such as the Medical Council of Canada Qualifying Examination Part II, a clinical skills objective structured clinical examination, have been permanently cancelled. The authors explore potential consequences of this cancellation including those that are inadvertent and undesirable. Future next steps for clinical skills assessment are explored.

"Clinical Teachers: Teaching Tips for the Busy Veterinary Team": Reflections on the Development of a Multimodal Resource for Veterinarians and Veterinary Nurses/ Technologists New to Clinical Teaching

Clinical practice-based training/work-integrated learning is an applied, social, and high-impact element of the veterinary curriculum. Within this context, students are learning on the job with clinician-educators who are carrying out their professional duties at the same time as supporting learning. To equip clinician-educators with role awareness and general teaching skills, it is recommended that all have access to basic teacher training. However, delivering this training can be challenging to organize and potentially costly when busy, time-poor clinician-educators are distributed across many geographical locations. This Teaching Tip shares our insights about developing and delivering a set of novel clinical teacher resources for veterinarians and veterinary nurses/technologists new to clinical teaching. The resources, underpinned by the principles of participatory design, integrate contemporary clinical educational theories with practical strategies and are interwoven with video clips capturing staff and student perspectives on key topics. While initially focused on creating just an online resource, we ultimately produced an A6 ring-bound booklet version and face-to-face workshops. In this article, we unpack considerations involved in committing to such a project and designing and creating the resources. We hope that this information may be of use to others when developing similar resources.

Competency-Based Assessment in Experiential Learning in Undergraduate Pharmacy Programmes: Qualitative Exploration of Facilitators' Views and Needs (ACTp Study)

Newly registered pharmacists will need to possess higher-level competencies and, in Great Britain, there is an expectation that assessments are undertaken during experiential learning (EL). The aim of this study was to explore the perceptions and educational needs of practice-based EL facilitators of student pharmacists, undertaking competency-based assessments during EL. Semi-structured one-on-one interviews were conducted with EL facilitators working in the community, hospital, and primary-care pharmacies. Data were thematically analysed. Fifteen facilitators were interviewed, and there were five from each site. There was general support for this role, but also anxiety due to the lack of knowledge about assessments and the repercussions on students. Benefits were that students would receive real-time feedback from workplace-based practitioners and facilitators would benefit from self-development. Challenges included additional workload and lack of consistency in marking. The majority agreed that clinical, professional, and communication skills could be assessed; however, a consensus was not reached regarding the tools, methods, and grading of assessments. The need for training and support were highlighted. A co-design method was proposed to ensure that the assessment methods and processes are accepted by all stakeholders. Training and resources should be tailored to the needs of facilitators.

Validity evidence for summative performance evaluations in postgraduate community pharmacy education

INTRODUCTION

Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.

METHODS

The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.

RESULTS

A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.

CONCLUSIONS

The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

Failure to fail: Fear of retribution or a response to neglecting the learner?

Rudland and Rennie comment on Swails et al.'s observation that staff are failing fewer learners, offering reflection on the impact neglecting a learner may have on educator decisions if the failure to fail phenomenon is considered.

Role of faculty characteristics in failing to fail in clinical clerkships

INTRODUCTION

In the context of competency-based medical education, poor student performance must be accurately documented to allow learners to improve and to protect the public. However, faculty may be reluctant to provide evaluations that could be perceived as negative, and clerkship directors report that some students pass who should have failed. Student perception of faculty may be considered in faculty promotion, teaching awards, and leadership positions. Therefore, faculty of lower academic rank may perceive themselves to be more vulnerable and, therefore, be less likely to document poor student performance. This study investigated faculty characteristics associated with low performance evaluations (LPEs).

METHOD

The authors analysed individual faculty evaluations of medical students who completed the third-year clerkships over 15 years using a generalised mixed regression model to assess the association of evaluator academic rank with likelihood of an LPE. Other available factors related to experience or academic vulnerability were incorporated including faculty age, race, ethnicity, and gender.

RESULTS

The authors identified 50 120 evaluations by 585 faculty on 3447 students between January 2007 and April 2021. Faculty were more likely to give LPEs at the midpoint (4.9%), compared with the final (1.6%), evaluation (odds ratio [OR] = 4.004, 95% confidence interval [CI] [3.59, 4.53]; p < 0.001). The likelihood of LPE decreased significantly during the 15-year study period (OR = 0.94 [0.90, 0.97]; p < 0.01). Full professors were significantly more likely to give an LPE than assistant professors (OR = 1.62 [1.08, 2.43]; p = 0.02). Women were more likely to give LPEs than men (OR = 1.88 [1.37, 2.58]; p 0.01). Other faculty characteristics including race and experience were not associated with LPE.

CONCLUSIONS

The number of LPEs decreased over time, and senior faculty were more likely to document poor medical student performance compared with assistant professors.

Tangrams: a simple visual tool for communicating the complexities of professionalism

Professionalism is vital for high quality healthcare and fundamental to health profession education. It is however complex, hard to define and can be challenging to teach, learn about and assess. We describe the development and use of an innovative visual tool, using a tangram analogy, to introduce and explore core professionalism concepts, which are often troublesome for both learners and educators. These include the hidden curriculum, capability, professional identity and the difference between unprofessionalism and high professional standards.  Understanding these concepts can help individuals to see professionalism differently, encourage faculty to design professionalism programmes which focus on professional excellence, support assessors to feel more confident in identifying and addressing underperformance and facilitate learners to appreciate the complexity and uncertainty inherent in professionalism and to become more alert to the hidden curriculum and its potential impact. We have used the tangram model to educate for professionalism in multiple contexts with learners and educators. Participants regularly report that it leads to a deeper understanding and important new insights around professionalism and helps them identify ways of changing their practice.  We believe this approach has relevance across the health professions and suggest ways it could be further developed to explore wider professionalism issues such as reflective practice, resilience and teamworking.

“Could You Work in My Team?”: Exploring How Professional Clinical Role Expectations Influence Decision-Making of Assessors During Exit-Level Medical School OSCEs

Decision-making in clinical assessment, such as exit-level medical school Objective Structured Clinical Examinations (OSCEs), is complex. This study utilized an empirical phenomenological qualitative approach with thematic analysis to explore OSCE assessors' perceptions of the concept of a “prototypical intern” expressed during focus group discussions. Topics discussed included the concept of a prototypical intern, qualities to be assessed, and approaches to clinical assessment decision-making. The thematic analysis was then applied to a theoretical framework (Cultural Historical Activity Theory—CHAT) that explored the complexity of making assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Ten Australasian medical schools were involved with 15 experienced and five less experienced assessors participating. Thematic analysis of the data revealed four major themes in relation to how the prototypical intern concept influences clinical assessors' judgements: (a) Suitability of marking rubric based on assessor characteristics and expectations; (b) Competence as final year student vs. performance as a prototypical intern; (c) Safety, trustworthiness and reliability as constructs requiring assessment and (d) Contradictions in decision making process due to assessor differences. These themes mapped well within the interaction between two proposed activity systems in the CHAT model: academic and clinical. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. Further research is needed to explore whether consensus on desirable intern qualities and their inclusion into OSCE marksheets decreases the cognitive load and increases the validity of assessor decision making.

Post graduate remediation programs in medicine: a scoping review

BACKGROUND

Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs.

METHODS

Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions.

RESULTS

The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs.

CONCLUSION

Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.

The shift from disbelieving underperformance to recognising failure: A tipping point model

CONTEXT

Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed.

METHODS

Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process.

RESULTS

The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm.

CONCLUSION

Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

How Perceived Burnout Alters Frontline Educators' Assessments in the Clinical Learning Environment

OBJECTIVE

This study explores frontline educators' experience with learner burnout in the clinical learning environment (CLE) and how perceived burnout may impact assessment.

METHODS

A survey was sent to 105 Pediatric Hospital Medicine faculty and fellows at 7 sites across the United States representing diverse CLEs. They were invited to participate in an 11-question web-based survey that was developed, edited, and revised in step-wise fashion. It consisted of 5-point Likert scale, forced-choice, and dichotomous questions. Data from the web-based survey were compiled for descriptive purposes and analyzed for trends.

RESULTS

The response rate was 53%. Most respondents (83%) perceive learner burnout at least once per year, but median comfort in identifying burnout was 3 (scale 1-5, 3 = neutral). Factors associated with comfort were education as primary niche (P = .01) and having wellness training (P = .045). In terms of the impact of perceived learner burnout impacts on assessment, 88% report impact on feedback sessions and 60% reported impact on summative assessment; most are more lenient. Stated belief in whether perceived burnout should or should not impact assessment had 60% sensitivity and 84% specificity in predicting whether it does.

CONCLUSIONS

Frontline educators routinely perceive learner burnout in the CLE and it often impacts educators' assessment of a learner. The discrepancy between the expected and actual impact may suggest amplification of known barriers to accurate assessment in the CLE. Comfort associated with wellness training suggests that interventions targeting frontline educators in the CLE may improve their ability to simultaneously support and assess learners.

How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?

OBJECTIVE

The aim of this study is to assess the quality of feedback provided to surgical trainees in the operating theatre, and to further investigate how trainees and trainers use workplace-based assessment in practice with regards to frequency and timing of assessments.

DESIGN

A retrospective study of all submitted Supervised Structured Assessments of Operative Performance (SSAOPs) from April 25, 2016 to February 2, 2021 was conducted.

SETTING

Surgical trainees in the Republic of Ireland across all national surgical training sites submitted SSAOPs through an online platform.

PARTICIPANTS

Assessments of operative competence (SSAOPs) from all Core Surgical Trainees (in their first two years of dedicated post-graduate surgical training) were included for analysis, regardless of surgical subspecialty. A total of 2294 assessments were submitted from April 25, 2016 to February 2, 2021 by 330 core surgical trainees and 379 surgeon assessors. Five hundred of these assessments were randomly selected and scored for quality of feedback using a modified "Task, Gap, Action (TGA)" framework.

RESULTS

Of all 2294 submitted assessments, 1905 (83.04%) were submitted in the latter 3 months of each rotation, and 803 (35%) were submitted in the last month. Only 51 of 270 (18.89%) of trainees in their first year and 33 of 236 trainees in their second year (13.98%) submitted more than the minimum required number of assessments (6 per year). Of 500 randomly selected assessments, 362 (72.4%) had documented written feedback. The mean 'Gap' and 'Action' scores were low, at 0.44/3 and 0.53/3 respectively.

CONCLUSIONS

Trainees do not submit more than the required number of operative workplace-based assessments. Assessments are submitted at the end of the trainee's rotation, limiting their formative value. The quality of written feedback is poor and could be improved significantly by encouraging a "Task," "Gap" and "Action" approach.

Facing the "Fear of Failure": Veterinary Students in Clinical Rotations

Failing a student is difficult for both educator and student, but administering a failing grade is critical for protecting and ensuring adequate learning for an unsafe student. The failure to fail clinical students has been commonly reported and explored among educators in the human health professions but has not been formally addressed in veterinary education. Forty-three participants attending the Veterinary Educators Collaborative symposium were surveyed concerning their attitudes and experiences failing clinical veterinary students. Results indicated that the failure to fail phenomenon exists among veterinary educators, as the majority of veterinary educators often felt reluctant and unprepared to fail a student on clinical rotations. The most common barriers to failing students were institutional culture and unsatisfactory assessor development or evaluation tools. Veterinary educators must face this fear of failure and explore strategies to overcome existing barriers that can ultimately transform student failure into success.

Tangrams: a simple visual tool for communicating the complexities of professionalism

Professionalism is vital for high quality healthcare and fundamental to health profession education. It is however complex, hard to define and can be challenging to teach, learn about and assess. We describe the development and use of an innovative visual tool, using a tangram analogy, to introduce and explore core professionalism concepts, which are often troublesome for both learners and educators. These include the hidden curriculum, capability, professional identity and the difference between unprofessionalism and high professional standards.  Understanding these concepts can help individuals to see professionalism differently, encourage faculty to design professionalism programmes which focus on professional excellence, support assessors to feel more confident in identifying and addressing underperformance and facilitate learners to appreciate the complexity and uncertainty inherent in professionalism and to become more alert to the hidden curriculum and its potential impact. We have used the tangram model to educate for professionalism in multiple contexts with learners and educators. Participants regularly report that it leads to a deeper understanding and important new insights around professionalism and helps them identify ways of changing their practice.  We believe this approach has relevance across the health professions and suggest ways it could be further developed to explore wider professionalism issues such as reflective practice, resilience and teamworking.

A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

The effect of gender dyads on the quality of narrative assessments of general surgery trainees

BACKGROUND

Prior studies have shown that gender can influence how learners are assessed and the feedback they receive. We investigated the quality of faculty narrative comments in general surgery trainee evaluation using trainee-assessor gender dyads.

METHODS

Narrative assessments of surgical trainees at the University of British Columbia were collected and rated using the McMaster Narrative Comment Rating Scale (MNCRS). Variables from the MNCRS were inputted into a generalized linear mixed model to explore the impact of gender dyads on the quality of narrative feedback.

RESULTS

2,469 assessments were collected. Women assessors tended to give higher-quality comments (p's < 0.05) than men assessors. Comments from men assessors to women trainees were significantly more positive than comments from men assessors to men trainees (p = 0.02). Men assessors also tended to give women trainees more reinforcing than corrective comments than to men trainees (p < 0.01).

CONCLUSIONS

There are significant differences in the quality of faculty feedback to trainees by gender dyads. A range of solutions to improve and reduce differences in feedback quality are discussed.