Assessment Question Characteristics Predict Medical Student Performance in General Pathology

Assessment of Candidates' Performance Pre- and Post-Adoption of Standard Setting in College Examinations between 2016 and 2023

BACKGROUND

Assessment drives candidates' learning and focuses on the main goals and objectives of the course. Over the years, determination of pass scores in medical education has evolved from the arbitrary 50% to the actual determination of the cut-off scores in the standard setting process in order to make a decision to pass/fail candidates presenting for licensure or credentialing purposes.

MATERIALS AND METHODS

The study aimed at assessing and comparing candidates' performance pre- and post-adoption of standard setting in the college. This was a comparative cross-sectional study with a time trend. Examination results of all candidates who registered and sat for examinations in the years 2016-2023 were included in the study. Secondary data from approved results of the various faculties from 2016 to 2023 were used. Data analysis was done using SPSS Version 26.

RESULTS

There was a statistically significant difference in the mean pass rates pre- and post-adoption of standard setting in the primary examinations (P < 0.001), Part I examinations (P = 0.002) and Part II examinations (P < 0.001) with a higher mean pass rate post-adoption of standard setting.

CONCLUSION

The increased pass rates in the primary, Part I and Part II examinations post-adoption of standard setting may suggest improved examination performance. There is, however, the need to assess the acquired competencies and skills of the candidates post-certification. The significantly higher mean pass rates post-adoption of standard setting are consequent on improvement in the quality and robustness of items/questions and of the examination processes, brought about by the training and retraining of faculty examiners which preceded the implementation.

Associations between item characteristics and statistical performance for paediatric medical student multiple choice assessments

Background: Multiple choice questions (MCQs) are commonly used in medical student assessments but often prepared by clinicians without formal education qualifications. This study aimed to inform the question writing process by investigating the association between MCQ characteristics and commonly used statistical measures of individual item quality for a paediatric medical term. Methods: Item characteristics and statistics for five consecutive annual barrier paediatric medical student assessments (each n=60 items) were examined retrospectively. Items were characterised according to format (single best answer vs. extended matching); stem and option length; vignette presence and whether required to answer the question, inclusion of images/tables; clinical skill assessed; paediatric speciality; clinical relevance/applicability; Bloom's taxonomy domain and item flaws. For each item, we recorded the facility (proportion of students answering correctly) and point biserial (discrimination). Results: Item characteristics significantly positively correlated (p<0.05) with facility were relevant vignette, diagnosis or application items, longer stem length and higher clinical relevance. Recall items (e.g., epidemiology items) were associated with lower facility. Characteristics significantly correlated with higher discrimination were extended matching question (EMQ) format, longer options, diagnostic and subspeciality items. Variation in item characteristics did not predict variation in the facility or point biserial (less than 10% variation explained). Conclusions: Our research supports the use of longer items, relevant vignettes, clinically-relevant content, EMQs and diagnostic items for optimising paediatric MCQ assessment quality. Variation in item characteristics explains a small amount of the observed variation in statistical measures of MCQ quality, highlighting the importance of clinical expertise in writing high quality assessments.

Evaluation of the quality of multiple-choice questions according to the students' academic level

BACKGROUND

One of the most important challenges in medical education is the preparation of multiple-choice questions able to discriminate between students with different academic level. Average questions may be very easy for students with good performance, reducing their discriminant power in this group of students. The aim of this study was to analyze if the discriminative power of multiple-choice questions is different according to the students' academic performance.

METHODS

We retrospectively analyzed the difficulty and discrimination indices of 257 multiple-choice questions used for the end of course examination of pathophysiology and analyzed whether the discrimination indices were lower in students with good academic performance (group 1) than in students with moderate/poor academic performance (group 2). We also evaluated whether case-based questions maintained their discriminant power better than factual questions in both groups of students or not. Comparison of the difficulty and discrimination indices between both groups was based on the Wilcoxon test.

RESULTS

Difficulty index was significantly higher in group 1 (median: 0.78 versus 0.56; P <  0.001) and discrimination index was significantly higher in group 2 (median: 0.21 versus 0.28; P <  0.001). Factual questions had higher discriminative indices in group 2 than in group 1 (median: 0.28 versus 0.20; P <  0.001), but discriminative indices of case-based questions did not differ significantly between groups (median: 0.30 versus 0.24; P = 0.296).

CONCLUSIONS

Multiple-choice question exams have lower discriminative power in the group of students with high scores. The use of clinical vignettes may allow to maintain the discriminative power of multiple-choice questions.

Outcomes of Remote Pathology Instruction in Student Performance and Course Evaluation

The coronavirus disease 2019 (COVID-19) pandemic has disrupted undergraduate medical education, including preclinical class-based courses, by requiring social distancing and essentially eliminating in-person teaching. The aim of this study was to compare student performance and satisfaction before and after implementation of remote instruction in a first-year introductory pathology course. Assessments (3 quizzes, 1 practical exam, and 1 final) were compared between courses given before (January 2020) and during (January 2021) the COVID-19 pandemic, in terms of mean scores, degree of difficulty, and item discrimination, both overall and across different question types. Students’ evaluations of the course (Likert scale-based) were also compared between the 2 years. Significantly higher mean scores were observed during remote instruction (compared to the prior, in-person year) on verbatim-repeated questions (94.9 ± 8.8 vs 89.4 ± 12.2; P = .002) and on questions incorporating a gross specimen image (88.4 ± 7.5 vs 84.4 ± 10.3; P = .007). The percentage of questions that were determined to be moderate/hard in degree of difficulty and good/very good in item discrimination remained similar between the 2 time periods. In the practical examination, students performed significantly better during remote instruction on questions without specimen images (96.5 ± 7.0 vs 91.2 ± 15.2; P = .004). Finally, course evaluation metrics improved, with students giving a higher mean rating value in each measured end point of course quality during the year of remote instruction. In conclusion, student performance and course satisfaction generally improved with remote instruction, suggesting that the changes implemented, and their consequences, should perhaps inform future curriculum improvements.