A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness

INTRODUCTION

Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances.

METHODS

We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity.

RESULTS

Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity.

CONCLUSIONS

The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study

INTRODUCTION

Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances.

METHODS AND ANALYSIS

The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data.

ETHICS AND DISSEMINATION

The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.

Complex hybridization dynamics between golden-winged and blue-winged warblers (Vermivora chrysoptera and Vermivora pinus) revealed by AFLP, microsatellite, intron and mtDNA markers

Blue-winged (Vermivora pinus) and golden-winged warblers (Vermivora chrysoptera) have an extensive mosaic hybrid zone in eastern North America. Over the past century, the general trajectory has been a rapid replacement of chrysoptera by pinus in a broad, northwardly moving area of contact. Previous mtDNA-based studies on these species' hybridization dynamics have yielded variable results: asymmetric and rapid introgression from pinus into chrysoptera in some areas and bidirectional maternal gene flow in others. To further explore the hybridization genetics of this otherwise well-studied complex, we surveyed variation in three nuclear DNA marker types--microsatellites, introns, and a panel of amplified fragment length polymorphisms (AFLPs)--with the goal of generating a multilocus assay of hybrid introgression. All markers were first tested on birds from phenotypically and mitochondrially pure parental-type populations from outside the hybrid zone. Searches for private alleles and assignment test approaches found no combination of microsatellite or intron markers that could separate the parental populations, but seven AFLP characters exhibited significant frequency differences among them. We then used the AFLP markers to examine the extent and pattern of introgression in a population where pinus-phenotype individuals have recently invaded a region that previously supported only a chrysoptera-phenotype population. Despite the low frequency of phenotypic hybrids at this location, the AFLP data suggest that almost a third of the phenotypically pure chrysoptera have introgressed genotypes, indicating the presence of substantial cryptic hybridization in the history of this species. The evidence for extensive cryptic introgression, combined with the lack of differentiation at other nuclear loci, cautions against hybrid assessments based on single markers or on phenotypic traits that are likely to be determined by a small number of loci. Considered in concert, these results from four classes of molecular markers indicate that pinus and chrysoptera are surprisingly weakly differentiated and that far fewer genetically 'pure' populations of chrysoptera may exist than previously assumed, two findings with broad implications for the conservation of this rapidly declining taxon.