Evaluation of the effect of items' format and type on psychometric properties of sixth year pharmacy students clinical clerkship assessment items

BACKGROUND

Examinations are the traditional assessment tools. In addition to measurement of learning, exams are used to guide the improvement of academic programs. The current study attempted to evaluate the quality of assessment items of sixth year clinical clerkships examinations as a function of assessment items format and type/structure and to assess the effect of the number of response choices on the characteristics of MCQs as assessment items.

METHODS

A total of 173 assessment items used in the examinations of sixth year clinical clerkships of a PharmD program were included. Items were classified as case based or noncase based and as MCQs or open-ended. The psychometric characteristics of the items were studied as a function of the Bloom's levels addressed, item format, and number of choices in MCQs.

RESULTS

Items addressing analysis skills were more difficult. No differences were found between case based and noncase based items in terms of their difficulty, with a slightly better discrimination in the latter. Open-ended items were easier, yet more discriminative. MCQs with higher number of options were easier. Open-ended questions were significantly more discriminative in comparison to MCQs as case based items while they were more discriminative as noncase based items.

CONCLUSION

Item formats, structure, and number of options in MCQs significantly affected the psychometric properties of the studied items. Noncase based items and open-ended items were easier and more discriminative than case based items and MCQs, respectively. Examination items should be prepared considering the above characteristics to improve their psychometric properties and maximize their usefulness.

Effectiveness of e-Learning in a Medical School 2.0 Model: Comparison of Item Analysis for Student-Generated vs. Faculty-Generated Multiple-Choice Questions

BACKGROUND

Early reports in the literature describe using student-generated questions as a method of student learning as well as augmenting question exam banks. Reports on the performance of student-generated questions versus faculty-generated questions, however, remain limited. This study aims to compare the question performance of student-generated versus faculty-generated multiple-choice questions (MCQ).

OBJECTIVES

To determine if student-generated questions using mobile audience response systems and online discussion boards have similar item discrimination scores as faculty-generated questions.

METHODS

A team-based learning session was used to create 113 student-generated multiple-choice questions (SGQs). A 20 question MCQ quiz was presented to a second year medical school class made of 10 randomly selected SGQs and 10 randomly selected faculty-generated multiple-choice questions (FGQs). Item analysis was performed on the test results.

RESULTS

The data showed no statistical difference in the point-biserial scores between the two groups (average point-biserial 0.31 students vs 0.36 faculty, p=0.14), with 90% of student-generated and 100% of faculty-generated questions meeting a cut-off of point-biserial score >0.2. Interestingly, student-generated questions were statistically more difficult than the faculty-generated questions (Item Difficulty score 0.46 students vs 0.69 faculty, p=0.003).

CONCLUSIONS

This study suggests that student-generated compared to faculty-generated MCQs have similar item discrimination scores, but are perhaps more difficult questions.