Evaluation of the effect of items' format and type on psychometric properties of sixth year pharmacy students clinical clerkship assessment items

BACKGROUND

Examinations are the traditional assessment tools. In addition to measurement of learning, exams are used to guide the improvement of academic programs. The current study attempted to evaluate the quality of assessment items of sixth year clinical clerkships examinations as a function of assessment items format and type/structure and to assess the effect of the number of response choices on the characteristics of MCQs as assessment items.

METHODS

A total of 173 assessment items used in the examinations of sixth year clinical clerkships of a PharmD program were included. Items were classified as case based or noncase based and as MCQs or open-ended. The psychometric characteristics of the items were studied as a function of the Bloom's levels addressed, item format, and number of choices in MCQs.

RESULTS

Items addressing analysis skills were more difficult. No differences were found between case based and noncase based items in terms of their difficulty, with a slightly better discrimination in the latter. Open-ended items were easier, yet more discriminative. MCQs with higher number of options were easier. Open-ended questions were significantly more discriminative in comparison to MCQs as case based items while they were more discriminative as noncase based items.

CONCLUSION

Item formats, structure, and number of options in MCQs significantly affected the psychometric properties of the studied items. Noncase based items and open-ended items were easier and more discriminative than case based items and MCQs, respectively. Examination items should be prepared considering the above characteristics to improve their psychometric properties and maximize their usefulness.

Quantitative analysis of single best answer multiple choice questions in pharmaceutics

INTRODUCTION

The purpose of this study was to: (1) analyze the quality of single best answer multiple choice questions (MCQs) used in pharmaceutics exams, (2) identify the correlation between difficulty index (DIF I), discriminating index (DI), and distractor efficiency (DE), and (3) understand the relationship between DIF I, DI, and DE and the number of MCQ answer options and their cognitive level.

METHODS

429 MCQs used in pharmaceutics exams were analyzed. The quality of the MCQs was evaluated using DIF I, DI, and DE. The number of answer options and the cognitive level tested by each item were evaluated. Relationships between DIF I, DI, and DE were measured using Pearson's correlations and t-tests.

RESULTS

DIF I showed a significant negative correlation with DI within questions that measured information recall. A significant negative correlation between DIF I and DI was observed in questions with four and five answer options regardless of the cognitive level measured. The highest DI values were found in moderate difficulty questions, while the worst DE was observed for the easiest questions. Questions that measured analytical and problem-solving abilities were more difficult than those measuring information recall. Questions with four and five answer options had excellent discrimination.

CONCLUSIONS

Single best answer MCQs are a valuable assessment tool capable of evaluating higher cognitive skills. Significant correlation between DIF I and DI can indicate the examination quality. Higher quality MCQs are constructed using four and five answer options.