Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

Promoting Longitudinal and Developmental Computer-Based Assessments of Clinical Reasoning: Validity Evidence for a Clinical Reasoning Mapping Exercise

PURPOSE

Clinical reasoning is vitally important for practitioners across the health professions. However, the assessment of clinical reasoning remains a significant challenge. Combined with other assessment methods, non-workplace-based assessment can increase opportunities to address multiple components of clinical reasoning, evaluate growth, and foster learning, but tools with validity evidence to assess clinical reasoning outside the workplace are scare. This study examined validity evidence for a novel clinical reasoning mapping exercise (CResME).

METHOD

Data include CResME performance scores from 120 third-year medical students at the University of Central Florida for 3 topics in May 2022. Each CResME was scored by 2 physician raters based on a scoring rubric that included a combined diagnosis and sequence score. Descriptive statistics were used to examine trends in scores. The authors gathered validity evidence for response process, internal structure, and relations to other variables.

RESULTS

The overall mean (SD) score across cases was 66 (29). Internal consistency reliability of cases (Cronbach α) ranged from 0.75 to 0.91. The Phi and G coefficients were 0.45 and 0.56, respectively. Students accounted for 10% of the total variance, indicating the ability to differentiate high and low clinical reasoning skills; the interaction between learner and case accounted for 8.1% of the variance, demonstrating case specificity. There was a moderate correlation between the overall CResME scores and the mean overall score of patient encounter notes from an objective structured clinical examination performed at the end of the third year (0.46; P = .001). Significant associations were also found between the CResME scores and subject exam scores.

CONCLUSIONS

The CResME can be used to facilitate the assessment of clinical reasoning, supporting the developmental progress of learners throughout the curriculum. Future research is needed to gather validity evidence for CResMEs with different learners across different settings and formats.

Assessing clinical reasoning skills following a virtual patient dizziness curriculum

OBJECTIVES

Dizziness is a common medical symptom that is frequently misdiagnosed. While virtual patient (VP) education has been shown to improve diagnostic accuracy for dizziness as assessed by VPs, trainee performance has not been assessed on human subjects. The study aimed to assess whether internal medicine (IM) interns after training on a VP-based dizziness curriculum using a deliberate practice framework would demonstrate improved clinical reasoning when assessed in an objective structured clinical examination (OSCE).

METHODS

All available interns volunteered and were randomized 2:1 to intervention (VP education) vs. control (standard clinical teaching) groups. This quasi-experimental study was conducted at one academic medical center from January to May 2021. Both groups completed pre-posttest VP case assessments (scored as correct diagnosis across six VP cases) and participated in an OSCE done 6 weeks later. The OSCEs were recorded and assessed using a rubric that was systematically developed and validated.

RESULTS

Out of 21 available interns, 20 participated. Between intervention (n=13) and control (n=7), mean pretest VP diagnostic accuracy scores did not differ; the posttest VP scores improved for the intervention group (3.5 [SD 1.3] vs. 1.6 [SD 0.8], p=0.007). On the OSCE, the means scores were higher in the intervention (n=11) compared to control group (n=4) for physical exam (8.4 [SD 4.6] vs. 3.9 [SD 4.0], p=0.003) and total rubric score (43.4 [SD 12.2] vs. 32.6 [SD 11.3], p=0.04).

CONCLUSIONS

The VP-based dizziness curriculum resulted in improved diagnostic accuracy among IM interns with enhanced physical exam skills retained at 6 weeks post-intervention.

A Team-based Learning Approach During Pediatric Clerkship to Promote Clinical Reasoning

BACKGROUND

Clinical reasoning skills are essential to medical practice. The problem lies in assuming junior medical students with limited experience develop clinical reasoning and decision-making skills passively and solely through clinical exposure. To prepare learners for independent practice and care for future patients, explicit teaching and assessment of clinical reasoning in low-stakes, collaborative learning environments are needed.

METHODS

The key-feature question (KFQs) format is an assessment approach that focuses on the reasoning and decision-making aspects of medical problems rather than knowledge recall. This report describes the development, implementation, and evaluation of a team-based learning (TBL) approach using KFQs to foster clinical reasoning in the third-year pediatric clerkship at our institution.

RESULTS

Over the first 2 years of implementation (2017-18, 2018-19), 278 students participated in TBL sessions. Mean individual student scores significantly improved in the group setting for both academic years (P < .001). Individual scores were moderately positively correlated with their total summative Objective Structured Clinical Examination score (r[275] = 0.51; P < .001). The correlation between individual score and their multiple-choice question examination was weaker at 0.29 (P < .001) but remained positive.

CONCLUSIONS

A TBL session using KFQs to both teach and assess clinical reasoning in clerkship students may help educators identify students with gaps in knowledge and/or reasoning. The next steps include developing and implementing individualized coaching opportunities, and expanding this approach within the undergraduate medical curriculum. Outcome measures to assess clinical reasoning in authentic patient encounters require further research and development.

Identifying the response process validity of clinical vignette-type multiple choice questions: An eye-tracking study

INTRODUCTION

Clinical vignette-type multiple choice questions (CV-MCQs) are widely used in assessment and identifying the response process validity (RPV) of questions with low and high integration of knowledge is essential. Answering CV-MCQs of different levels of knowledge application and integration can be understood from a cognitive workload perspective and this can be identified by using eye-tracking. The aim of the pilot study was to identify the cognitive workload and RPV of CV-MCQs of different levels of knowledge application and integration by the use eye-tracking.

METHODS

Fourteen fourth-year medical students answered a test with 40 CV-MCQs, which were equally divided into low-level and high-level complexity (knowledge application and integration). Cognitive workload was measured using screen-based eye tracking, with the number of fixations and revisitations for each area of interest.

RESULTS

We found a higher cognitive workload for high-level complexity (M = 121.74) compared with lower-level complexity questions (M = 51.94) and also for participants who answered questions incorrectly (M = 94.31) compared with correctly (M = 79.36).

CONCLUSION

Eye-tracking has the potential to become a useful and practical approach for helping to identify the RPV of CV-MCQs. This approach can be used for improving the design and development of CV-MCQs, and to provide feedback to inform teaching and learning.[Box: see text].

COVID-19 : Quelques pistes pour un nouvel environnement d’enseignement et d’apprentissage en contexte de mise à distance des enseignants et des étudiants en médecine

Contexte et problématique :La pandémie de coronavirus a affecté le système éducatif du monde entier. Les stages cliniques des étudiants en médecine sont majoritairement suspendus. Il existe une incertitude quant à la durée de la situation. De même il est difficile de prédire si des situations comparables vont se reproduire dans le futur. Pour garantir la continuité pédagogique, il est nécessaire de proposer d’autres stratégies pédagogiques en préservant au maximum l’authenticité clinique.Objectifs :Cet article vise à : (1) fournir quelques éléments de réflexion pour guider les enseignants dans ce nouvel environnement d’apprentissage ; (2) présenter différentes modalités d’enseignement à distance disponibles, leurs avantages et les enjeux, en s’appuyant sur la littérature ; (3) illustrer la mise en œuvre et l’évaluation d’une telle stratégie à partir d’un retour d’expérience concernant un module électif offert aux étudiants de deuxième cycle en stage dans le service d’endocrinologie.Résultats et conclusion : Les résultats préliminaires suggèrent une satisfaction globale des étudiants et le souhait de poursuivre ces séances dans le futur. Les enseignants et les étudiants doivent réussir à trouver les avantages et les enjeux potentiels des nouvelles technologies non seulement dans l’enseignement, mais également dans le domaine du diagnostic et celui de soins.