The influence of candidates' physical attributes on assessors' ratings in clinical practice

The influence of candidates' race on examiners' ratings in standardised assessments of clinical practice

PURPOSE

Delivering fair and reliable summative assessments in medical education assumes examiner decision making is devoid of bias. We investigated whether candidate racial appearances influenced examiner ratings in undergraduate clinical exams.

METHODS

We used an internet-based design. Examiners watched a randomised set of six videos of three different white candidates and three different non-white (Asian, black and Chinese) candidates taking a clinical history at either fail, borderline or pass grades. We compared the median and interquartile range (IQR) of the paired difference between scores for the white and non-white candidates at each performance grade and tested for statistical significance.

RESULTS

160 Examiners participated. At the fail grade, the black and Chinese candidates scored lower than the white candidate, with median paired differences of -2.5 and -1 respectively (both p < 0.001). At the borderline grade, the black and Chinese candidates scored higher than the white candidate, with median paired differences of +2 and +3, respectively (both p < 0.001). At the passing grade, the Asian candidate scored lower than the white candidate (median paired difference -1, p < 0.001).

CONCLUSION

The racial appearance of candidates appeared to influence the scores awarded by examiners, but not in a uniform manner.

Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions

INTRODUCTION

Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts.

METHODS

We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics.

RESULTS

We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs.

DISCUSSION

Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices.

The influence of candidates' physical attributes on patient ratings in simulated assessments of clinical practice

BACKGROUND

We have previously shown that clinical examiners' scoring is not negatively impacted when a candidate has a tattoo, unnatural hair colour, or a regional accent. We investigated whether these physical attributes in exam candidates impact patient scoring.

METHODS

Simulated/real patients were randomly assigned to watch five videos of simulated candidate performances of a cranial nerve examination: clear fail, borderline, good, 'clear pass' without an attribute, and 'clear pass' with one of the attributes (tattoo, purple hair, accent). Participants scored domains of communication and professionalism. We compared scores for the clear pass candidates with and without attributes.

RESULTS

One hundred and eighty three patients participated. The total scores for the candidates with tattoos and purple hair were higher than the candidate with no physical attribute (p < 0.001). For the candidate with a Liverpool English accent no difference was identified (p = 0.120).

CONCLUSIONS

The presence of certain physical attributes (tattoos or purple hair) was associated with higher scores given by patients to candidates in a simulated physical examination station.

[Contribution and limits of "OSCE", "long-case" and "global end-of-placement marking" as end-of-rotation assessment methods. Experience from two internal medicine wards]

INTRODUCTION

During placements, there is an opportunity to learn clinical skills and to assess their application. However, it represents two different goals. The validity of an end-of-placement assessment is questionable, as the medical competency is contextual. We decided to evaluate the contribution and limits of different assessment modalities as an end-of-placement assessment.

MATERIAL AND METHODS

Internal medicine clerks were assessed using the Mini-Cex grid by a structured objective clinical examination (OSCE), a long-case clinical examination (LCE) and a global end-of-placement marking (GEPM). Following these evaluations, students and teachers fulfilled an open questionnaire.

RESULTS

In 2021, 41 students and 16 teachers participated in the study. Physical examination was evaluated in 0%, 97% et 76% of cases during OSCE, LCE and GEPM, respectively; teaching skills were assessed for 100, 42 et 49% of students in OSCE, LCE and GEPM, respectively. As compared to OSCE, there was a perceived superiority of LCE regarding its formative value (P=0.07 and P=0.03) and its summative value (P=0.0007 and P=0.02), for students and teachers, respectively. Qualitative analysis highlights the breadth of clinical skills that could be assessed during OSCE stations. Integration into a team was an additional skill that could specifically be assessed during GEPM. GEPM could also take into account the progress made during placement.

CONCLUSION

Despite its subjectivity, LCE seemed to be the preferred modality for an end-of-rotation assessment.