Item Analysis of Multiple Choice Questions at the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain

An assessment of preclinical removable prosthodontics based on multiple-choice questions: Stakeholders' perceptions

PURPOSE

Item analysis of multiple-choice questions (MCQs) is an essential tool for identifying items that can be stored, revised, or discarded to build a quality MCQ bank. This study analyzed MCQs based on item analysis to develop a pool of valid and reliable items and investigate stakeholders' perceptions regarding MCQs in a written summative assessment (WSA) based on this item analysis.

METHODS

In this descriptive study, 55 questions each from 2016 to 2019 of WSA in preclinical removable prosthodontics for fourth-year undergraduate dentistry students were analyzed for item analysis. Items were categorized according to their difficulty index (DIF I) and discrimination index (DI). Students (2021-2022) were assessed using this question bank. Students' perceptions of and feedback from faculty members concerning this assessment were collected using a questionnaire with a five-point Likert scale.

RESULTS

Of 220 items when both indices (DIF I and DI) were combined, 144 (65.5%) were retained in the question bank, 66 (30%) required revision before incorporation into the question bank, and only 10 (4.5%) were discarded. The mean DIF I and DI values were 69% (standard deviation [Std.Dev] = 19) and 0.22 (Std.Dev = 0.16), respectively, for 220 MCQs. The mean scores from the questionnaire for students and feedback from faculty members ranged from 3.50 to 4.04 and from 4 to 5, respectively, indicating that stakeholders tended to agree and strongly agree, respectively, with the proposed statements.

CONCLUSION

This study assisted the prosthodontics department in creating a set of prevalidated questions with known difficulty and discrimination capacity.

Item analysis: the impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items

BACKGROUND

Distractor efficiency (DE) of multiple-choice questions (MCQs) responses is a component of the psychometric analysis used by the examiners to evaluate the distractors' credibility and functionality. This study was conducted to evaluate the impact of the DE on the difficulty and discrimination indices.

METHODS

This cross-sectional study was conducted from April to June 2023. It utilizes the final exam of the Principles of Diseases Course with 45 s-year students. The exam consisted of 60 type A MCQs. Item analysis (IA) was generated to evaluate KR20, difficulty index (DIF), discrimination index (DIS), and distractor efficiency (DE). DIF was calculated as the percentage of examinees who scored the item correctly. DIS is an item's ability to discriminate between higher and lower 27% of examinees. For DE, any distractor selected by less than 5% is considered nonfunctional, and items were classified according to the non-functional distractors. The correlation and significance of variance between DIF, DI, and DE were evaluated.

RESULTS

The total number of examinees was 45. The KR-20 of the exam was 0.91. The mean (M), and standard deviation (SD) of the DIF of the exam was 37.5(19.1), and the majority (69.5%) were of acceptable difficulty. The M (SD) of the DIS was 0.46 (0.22), which is excellent. Most items were excellent in discrimination (69.5%), only two were not discriminating (13.6%), and the rest were of acceptable power (16.9%). Items with excellent and good efficiency represent 37.3% each, while only 3.4% were of poor efficiency. The correlation between DE and DIF (p = 0.000, r= -0.548) indicates that items with efficient distractors (low number of NFD) are associated with those having a low difficulty index (difficult items) and vice versa. The correlation between DE and DIS is significantly negative (P = 0.0476, r=-0.259). In such a correlation, items with efficient distractors are associated with low-discriminating items.

CONCLUSIONS

There is a significant moderate negative correlation between DE and DIF (P = 0.00, r = -0.548) and a significant weak negative correlation between DE and DIS (P = 0.0476, r = -0.259). DIF has a non-significant negative correlation with DIS (P = 0.7124, r = -0.0492). DE impacts both DIF and DIS. Items with efficient distractors (low number of NFD) are associated with those having a low difficulty index (difficult items) and discriminating items. Improving the quality of DE will decrease the number of NFDs and result in items with acceptable levels of difficulty index and discrimination power.

A comparison of the assessments used in campus-based years at the College of Medicine, Imam Abdulrahman Bin Faisal University, Saudi Arabia

STUDY PURPOSE

Multiple assessment tools are used to assess future doctors' knowledge, clinical skills, and professional attitudes. In the present research, the difficulty level and discriminating ability of different types of written and performance-based assessments designed to measure the knowledge and competency of medical students were compared.

METHODS

The assessment data of 2nd & 3rd-year medical students (the academic year 2020-2021) in the College of Medicine at Imam Abdulrahman Bin Faisal University (IAU) were retrospectively reviewed. Based on end-of-the-year overall grades, students were divided into high and low scorers. Both groups were compared by independent sample t-test for their mean scores achieved in each type of assessment. Difficulty level and discriminating ability of the assessments were also explored. MS Excel and Statistical Package for Social Sciences (SPSS version 27) were used for analysis. Area under the curve was calculated through ROC analysis. A p-value <0.05 was believed significant.

RESULTS

In each type of written assessment, the high scorer group achieved significantly higher scores compared to the low scorers. Among performance-based assignments (except the PBLs), scores did not differ significantly between high and low scorers. The difficulty level of performance-based assessments was "easy" whereas it was "moderate" for written assessments (except the OSCE). The discriminating ability of performance-based assessments was "poor" whereas it was "moderate/excellent" for written assessments (except the OSCE).

CONCLUSION

Our study results indicate that written assessments have excellent discriminatory ability. However, performance-based assessments are not as difficult and discriminatory as written assessments. The PBLs are relatively discriminatory among all performance-based assessments. Key messages What is already known on this topic At Imam Abdulrahman Bin Faisal University, written and performance-based assessments both are graded on criterion-referenced scales. The student's grades at the end of the year are an aggregate of his/her scores in written and performance-based assessments. What this study adds Our study results show that performance-based assessments are not as difficult and discriminatory in differentiating between high and low scorers as written assessments. How this study might affect research, practice or policy Performance-based assessments should be made a hurdle exam (pass or fail) for the students to move to the next level, or students must pass each assessment component (written and performance-based) separately.

Best Practices of the World Health Organization Collaborating Centres (WHOCCs) in the Eastern Mediterranean Region

BACKGROUND

World Health Organization Collaborating Centres (WHOCCs) cooperate with the WHO on a range of strategic areas such as nursing, nutrition, mental health, chronic diseases, education, and health technologies, depending on their speciality areas. As of 2021, WHO has 47 CCs in the Eastern Mediterranean Region (EMR) collaborating on diverse areas. Four CCs in the EMR located in Egypt, Kingdom of Bahrain, Sudan, and Pakistan focus primarily on medical education (ME).

OBJECTIVE

The objective of this review of the literature is to describe the best practices in ME based on published research from the four WHOCCs in EMR. The secondary objective is to classify them based on the level of Kirkpatrick's model (KM) of educational outcomes.

METHODS

The contributions of WHOCCs are categorised in to five domains namely "Curriculum Development and Course Design", "Student Assessment", "Quality, Accreditation, and Program Evaluation", "Teaching and Learning" and "Innovation in Medical Education". Initial extraction yielded 96 articles for review, while the second level of analysis reduced the number of publications to 37 based on the date of publication within the last 5 years. Numerous best practices in ME emerged from the recently published works of these WHOCCs in the areas of learning and teaching, curriculum development, innovations in medical education, quality, and assessments in medical education. Literature from the WHOCCs on assessment and curriculum design are limited, possibly indicating opportunities for additional research.

CONCLUSION

The researchers conclude that the WHOCCs in the EMR show transformational impact on all principal areas of research and at multiple levels.

Quality of multiple-choice questions in medical internship qualification examination determined by item response theory at Debre Tabor University, Ethiopia

BACKGROUND

Assessment of cognitive competence is a major element of the internship qualification exam in undergraduate medical education in Ethiopia. Assessing the quality of exam items can help to improve the validity of assessments and assure stakeholders about the accuracy of the go/no decision to the internship. However, we know little about the quality of exam items utilized to ascertain fitness to join the medical internship. Therefore, this study aimed to analyze the quality of multiple-choice questions (MCQs) of the qualification exam administered to final-year medical students at Debre Tabor University (DTU), Ethiopia.

METHODS

A psychometric study was conducted to assess the qualities of 120 randomly selected MCQs and 407 distractors. Item characteristics were estimated using the item response theory (IRT) model. T-test, one-way ANOVA, and chi-square tests were run to analyze the univariate association between factors. Pearson's correlation test was done to determine the predictive validity of the qualification examination.

RESULT

Overall, 16, 51, and 33% of the items had high, moderate, and low distractor efficiency, respectively. About two-thirds (65.8%) of the items had two or more functioning distractors and 42.5% exhibited a desirable difficulty index. However, 77.8% of items administered in the qualification examination had a negative or poor discrimination index. Four and five option items didn't show significant differences in psychometric qualities. The qualification exam showed a positive predictive value of success in the national licensing examination (Pearson's correlation coefficient = 0.5).

CONCLUSIONS

The psychometric properties of the medical qualification exam were inadequate for making valid decisions. Five option MCQs were not better than four options in terms of psychometric qualities. The qualification examination had a positive predictive validity of future performance. High-stakes examination items must be properly created and reviewed before being administered.

An initial preparation for human cadaveric dissection ameliorates the associated mental distress in students

It is universally recognized that cadaveric dissection is an essential part of anatomy training. However, it has been reported to induce mental distress in some students and impair their intrinsic motivation (IM) to study. One of the postulated reasons for this behavior is the lack of adequate information and preparation of students for cadaveric dissection. Therefore, it is hypothesized that providing relevant information prior to cadaveric dissection will ameliorate the mental distress, enhance the IM of students, and improve their academic performance. A cohort of occupational therapy students enrolled in an anatomy course were psychologically prepared for cadaveric dissection. Students were provided with a curated list of YouTube videos and peer-reviewed journal articles related to cadaveric dissection prior to the commencement of the anatomy course. All students were also required to attend an oral presentation immediately before commencing dissection. The control group included students who had not been provided with any resources in preparation for cadaveric dissection. Compared to the control group, students who had been prepared demonstrated better quality of cadaveric dissection, improved academic performance, reported less mental distress and greater IM. Moreover, students reported the oral presentation to be most relevant and journal articles to be least useful in their preparation. Therefore, this is an effective approach in the amelioration of mental distress and improvement of performance in anatomy students. Consequently, this study represents a paradigm shift in the pedagogy of anatomy, and could represent a vital element in the evolution of a revitalized anatomy curriculum.

Auditing RadExam: Employing Psychometrics to Improve Exam Quality

INTRODUCTION

RadExam is a question item and exam database jointly developed by the Association of Program Directors in Radiology and the American College of Radiology to provide formative resident assessment, offering performance metrics benchmarked against institutional and national resident performance. Beyond resident performance, data is available on question and exam performance. Despite considerable investment in the education and training of its question writers and editors and meticulous attention to current psychometrically validated methods, it was anticipated a minority of exam questions would still perform poorly. Audits were performed to identify these questions, identify reasons for poor performance, and modify or replace so-affected questions. Exam performance was also assessed.

METHODS

Two audits were performed, the first after the February-May 2018 RadExam pilot phase, and the second nearly 1 year after the full implementation of RadExam. In each audit, RadExam subspecialty editors evaluated all exam questions and exams using statistical data: question and test number of administrations, question p value, question Discrimination Index (DI), question Bloom's taxonomy learning level, exam P-value, and the number of image-based questions in each exam. Identified questions were modified or removed and replaced.

RESULTS

Audit 1 was performed after the administration of 3114 exams comprised of 2520 questions administered across 100 residency programs. Audit 1 identified 617 questions with DI <0.1 and 565 questions with unacceptable P-values, all of which were modified or replaced. Audit 2 was performed after the administration of 16,416 exams, comprised of 2,507 questions. Audit 2 identified 229 questions with DI <0.1 and 290 questions with unacceptable P-values, representing a 49.1% decrease in total flagged questions compared to Audit 1. Statistically significant decreases were seen in questions with both DI and P-values outside of the desired range across nearly all subspecialties.

CONCLUSION

The positive impact of our audit system on question and exam performance was reflected in a significant decrease in the number of questions flagged and improved overall exam performance in Audit 2. This illustrates the positive impact of Audit 1.

Development and Validation of a Survey to Evaluate Mental Health Knowledge: The Case of Indonesian Pregnant Women

BACKGROUND: Internationally, many instruments have been designed to evaluate mental health knowledge; however, in pregnant women is very limited. AIM: Therefore, this study aimed to develop and validate a survey to measure the mental health knowledge of pregnant women. METHODS: In this cross-sectional study, 13 midwives attended the FGD and 10 pregnant women were invited for in-depth interviews to develop an item pool. The content validity was carried out by a panel of 6 experts. The face validity was performed with 5 pregnant women. Next, the construct validity test involved 150 pregnant women who were selected by stratified sampling from 13 public health centers in Surakarta, Indonesia. Analyses were conducted to check content validity, face validity, construct validity, internal consistency reliability, difficulty index, and exploratory factor analysis. RESULTS: A final 20-item Mental Health Knowledge Scale (MHKS) has a content validity index of 0.97 and a correlation value per item greater than the r-table (i.e., 0.1603). In addition, the MHKS has a Kuder–Richardson 20 reliability coefficient of 0.717. Furthermore, the difficulty index ranged from 0.39 to 0.82 which was considered in the good and acceptable category. Construct validity was confirmed using exploratory factor analysis KMO = 0.713, Bartlett’s test p < 0.001. CONCLUSION: Based on the findings, the final version of the MHKS was considered a valid and reliable tool. The instrument can be applied to measure the understanding of pregnant women about pregnancy depression. Further studies require adjustment items to other participants regarding mental health knowledge.

Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-sectional study

Purpose

Multiple-choice questions (MCQs) are useful in assessing student performance, covering a wide range of topics in an objective way. Its reliability and validity depend upon how well it is constructed. Defective Item detected by item analysis must be looked for item writing flaws and optimized. The aim of this study was to evaluate the MCQs for difficulty levels, discriminating power with functional distractors by item analysis, analyze poor items for writing flaws, and optimize.

Methods

This was a prospective cross-sectional study involving 120 MBBS students writing formative assessment in Ophthalmology. It comprised 40 single response MCQs as a part of 3-h paper for 20 marks. Items were categorized according to their difficulty index, discrimination index, and distractor efficiency with simple proportions, mean, standard deviation, and correlation. The defective items were analyzed for proper construction and optimized.

Results

The mean score of the study group was 13.525 ± 2.617. Mean difficulty index, discrimination index, and distractor efficiency were 53.22, 0.26, and 78.32, respectively. Among 40 MCQs, twenty-five MCQs did not have non-functioning distractor; 7 had one, 5 had two, and 3 had three. Of the 20 defective items, 17 were optimized and added to the question bank, two were added without modification, and one was dropped.

Conclusion

Item analysis is a valuable tool in detecting poor MCQs, and optimizing them is a critical step. The defective items identified should be optimized and not dropped so that the content area covered by the defective item is not kept of the assessment.

Psychometric Characteristics of Oral Pathology Test Items in the Dental Hygiene Curriculum-A Longitudinal Analysis

As the landscape of oral healthcare and the delivery of services continue to undergo change, the dental hygienist plays an increasing role in assisting dentists with oral diagnosis and preventive strategies. Hence, the dental hygiene curriculum standards require biomedical science instructions, including general and oral pathology. Student learning and cognitive competencies are often measured using multiple-choice questions (MCQs). The objectives of this study were to perform a longitudinal analysis of test items and to evaluate their relation to the absolute grades of the oral pathology course in the dental hygiene curriculum. A total of 1033 MCQs covering different concepts of oral pathology administered from 2015 through 2019 were analyzed for difficulty and discriminatory indices, and the differences between the years were determined by one-way ANOVA. Test reliability as determined by the average KR-20 value was 0.7 or higher for each exam. The mean difficulty index for all exams was 0.73 +/− 0.05, and that of the discriminatory index was 0.33 +/− 0.05. Wide variations were observed in the discriminatory indices of test items with approximately the same difficulty index, as well as in the grade distribution in each cohort. Furthermore, longitudinal data analyses identified low achieving cohorts amongst the groups evaluated for the same knowledge domain, taught with the same instruction, and using similar test tools. This suggest that comparative analyses of tests could offer feedback not only on student learning attributes, but also potentially on the admission processes to the dental hygiene program.

Experiences from Two Ways of Integrating Pre- and Post-course Multiple-choice Assessment Questions in Educational Events for Surgeons

To examine how to optimise the integration of multiple-choice questions (MCQs) for learning in continuing professional development (CPD) events in surgery, we implemented and evaluated two methods in two subspecialities over multiple years. The same 12 MCQs were administered pre- and post-event in 66 facial trauma courses. Two different sets of 10 MCQs were administered pre- and post-event in 21 small animal fracture courses. We performed standard psychometric tests on responses from participants who completed both the pre- and post-event assessment. The average difficulty index pre-course was 57% with a discrimination index of 0.20 for small animal fractures and 53% with a discrimination index of 0.15 for facial trauma. For the majority of the individual MCQs, the scores were between 30%-70% and the discrimination index was >0.10. The difficulty index post-course increased in both groups (to 75% and 62%). The pre-course MCQs resulted in an average score in the expected range for both formats suggesting they were appropriate for the intended level of difficulty and an appropriate pre-course learning activity. Post-course completion resulted in increased scores with both formats. Both delivery methods worked well in all regions and overall quality depends on applying a solid item development and validation process.

Changes in the learning styles and approaches of students following incorporation of drawing during cadaveric dissection

The teaching of anatomy is challenging due to the constraints of material and personnel resources. Research has established that the learning preferences of students are malleable and determined by the requirements of the course. Further, drawing has been reported to aid learning in anatomy by facilitating problem solving and reducing the cognitive overload in students. Considering these issues, the aims of the study were to investigate (a) if positive changes occur in the learning styles and approaches following the incorporation of drawing during cadaveric dissection, and (b) whether they are associated with improved learning outcomes. One cohort of students in an anatomy course received training in creating scientific drawings from dissected human cadavers, while two cohorts of students did not receive such training. The learning preferences of students and their final examination grades were assessed at the commencement and conclusion of the course. Majority of student who had training in drawing transitioned from being bimodal, to trimodal or quadrimodal learners. This was associated with efficient learning approaches and a significant (p < .05) improvement in learning outcomes in these students. There were no changes in any parameters in students who had not received training in drawing. Therefore, the modulation of learning preferences of students through drawing is a pragmatic approach in anatomy teaching.

Knowledge of the care of wounded patients: evidence of validity of an instrument

OBJECTIVE

to present evidence of validity of an instrument aimed at evaluating the knowledge of nursing students about the care of the wounded patient, according to the difficulty and discrimination indexes of the items.

METHOD

methodological study conducted in a nursing higher education institution with 117 undergraduate students and 38 professionals from a research group with experience in the area of wounds. For data collection, a questionnaire with 10 multiple choice questions was applied before and after classes on wounds.

RESULTS

most of the questions presented low level of difficulty and inefficient discrimination index, requiring a revision of the instrument. After two review stages, the difficulty and discrimination indexes of the instruments improved.

CONCLUSION

an instrument with better evidence of validity was obtained. However, it still requires refinement for later revalidation in the same population.

Evaluating the Quality of Multiple Choice Question in Paediatric Dentistry Postgraduate Examinations

Objectives

This study aimed to evaluate the quality of multiple choice question (MCQ) items in two postgraduate paediatric dentistry (PD) examinations by determining item writing flaws (IWFs), difficulty index (DI) and cognitive level.

Methods

This study was conducted at Mohamed Bin Rashid University of Medicine and Health Sciences, Dubai, UAE. Virtual platform-based summative versions of the general paediatric medicine (GPM) and prevention of oral diseases (POD) examinations administered during the second semester of the 2017–2018 academic year were used. Two PD faculty members independently reviewed each question to assess IWFs, DI and cognitive level.

Results

A total of 185 single best answer MCQs with 4–5 options were analysed. Most of the questions (81%) required information recall, with the remainder (19%) requiring higher levels of thinking and data explanation. The most common errors among IWFs were the use of “except” or “not” in the lead-in, tricky or unfocussed stems and opportunities for students to use convergence strategies. There were more IWFs in the GPM than the POD examination, but this was not statistically significant (P = 0.105). The MCQs in the GPM and POD examination were considered easy since the mean DIs (89.1% ± 8.9% and 76.5% ± 7.9%, respectively) were more than 70%.

Conclusion

Training is an essential element of adequate MCQ writing. A general comprehensive review of all programme’s MCQs is needed to emphasise the importance of avoiding IWFs. A faculty development programme is recommended to improve question-writing skills in order to align examinations with programme learning outcomes and enhance the ability to measure student competency through questions requiring higher level thinking.