Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment

The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system?

Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.

The Power of One Word to Paint a Halo or a Horn: Demonstrating the Halo Effect in Learner Handover and Subsequent Evaluation

PURPOSE

Learner evaluation based upon direct observation is a cornerstone of modern competency-based medical education. Learner handover has become a widely accepted practice. Cognitive attribution bias is a potential threat to the validity of learner evaluation following learner handover.

METHOD

In this 2 x 2 (learner handover: halos/horns x learner gender: man/woman) factorial, nonequivalent comparison group experimental design, registered attendees at a national medical education conference watched 1 of 2 videos (depicting a woman learner or man learner) of simulated learner-patient encounters. Data were collected on April 30 and May 1, 2022. Participants received learner handover conditioning before watching the video. The conditioning was randomized to suggest the learner they were about to watch was either "above-average" (halos) or "below-average" (horns). Following the video, participants completed an evaluation form.

RESULTS

Participants rated the learner in a video encounter preceded by a horns statement significantly lower than the learner in a video encounter preceded by a halo statement, F (1,65) = 10.15, P < .01, η 2 = .14, horns mean adj = 12.49 (CI 11.34, 13.63), halo mean adj = 15.10 (CI 13.93, 16.28). This represented a scoring difference of 2.61 points on a 30-point scale. More years of teaching experience was negatively associated with the score, F (1,65) = 13.44, P < .001, η 2 = .17.

CONCLUSIONS

Learner conditioning differing by a single word, labeling a learner as either "above-average" or "below-average" resulted in a large difference in scoring by medical educators.

Effects of informal learner handover in clinical dental education

BACKGROUND

It is currently under discussion whether Learner Handovers (LH) are beneficial, disadvantageous, or useful in Health Professions Education. Research has not been conducted to determine the extent of existing informal learner handover (ILH) through faculty discussions. In addition to providing stakeholders with added context, examining the nature of ILH may also provide insight into the bias associated with Learner Handover.

METHODS

Transcripts from a series of semi-structured Focus Group Discussions (FGDs) and interviews (from January to March 2022) were iteratively reviewed to identify relevant patterns and correlations. The study involved the voluntary participation of 16 active clinical dental faculty members with a variety of designations. We did not discard any opinions.

RESULTS

It was found that ILH had a mild impact on students' training. ILH effects can be categorized into four key areas: (1) faculty behavior with students, (2) faculty expectations from students, 3) teaching approach, and 4) faculty feedback practices. Furthermore, five additional factors were identified as having a greater influence on ILH practices.

CONCLUSIONS

In clinical dental training, ILH has a minor effect on faculty-student interactions. Faculty perceptions and ILH are strongly influenced by other factors contributing to the student's 'academic reputation. As a result, student-faculty interactions are never free of prior influences, so stakeholders need to take them into consideration when creating a formal LH.

Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study

INTRODUCTION

Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances.

METHODS AND ANALYSIS

The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data.

ETHICS AND DISSEMINATION

The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.