Evaluation of Cognitive levels and Item writing flaws in Medical Pharmacology Internal Assessment Examinations

Postexamination item analysis of undergraduate pediatric multiple-choice questions exam: implications for developing a validated question Bank

INTRODUCTION

Item analysis (IA) is widely used to assess the quality of multiple-choice questions (MCQs). The objective of this study was to perform a comprehensive quantitative and qualitative item analysis of two types of MCQs: single best answer (SBA) and extended matching questions (EMQs) currently in use in the Final Pediatrics undergraduate exam.

METHODOLOGY

A descriptive cross-sectional study was conducted. We analyzed 42 SBA and 4 EMQ administered to 247 fifth-year medical students. The exam was held at the Pediatrics Department, Qena Faculty of Medicine, Egypt, in the 2020-2021 academic year. Quantitative item analysis included item difficulty (P), discrimination (D), distractor efficiency (DE), and test reliability. Qualitative item analysis included evaluation of the levels of cognitive skills and conformity of test items with item writing guidelines.

RESULTS

The mean score was 55.04 ± 9.8 out of 81. Approximately 76.2% of SBA items assessed low cognitive skills, and 75% of EMQ items assessed higher-order cognitive skills. The proportions of items with an acceptable range of difficulty (0.3-0.7) on the SBA and EMQ were 23.80 and 16.67%, respectively. The proportions of SBA and EMQ with acceptable ranges of discrimination (> 0.2) were 83.3 and 75%, respectively. The reliability coefficient (KR20) of the test was 0.84.

CONCLUSION

Our study will help medical teachers identify the quality of SBA and EMQ, which should be included to develop a validated question bank, as well as questions that need revision and remediation for subsequent use.

Should multiple-choice questions get the SAQ? Development of a short-answer question writing rubric

INTRODUCTION

Short-answer questions (SAQs) are often used to assess pharmacy student competency. However, the literature lacks guidance on SAQ development strategies, resulting in varying practices between SAQ writers. Understanding student and faculty perceptions of what constitutes a high-quality SAQ can identify best practices for SAQ development.

METHODS

We surveyed second-year pharmacy students at the University of California San Francisco (UCSF) to assess their perceptions of SAQs. Likert-type data were descriptively analyzed, and open-ended responses were analyzed using thematic analysis; we used these results to draft an initial SAQ checklist. We then conducted focus groups of UCSF pharmacy faculty to explore their experiences writing SAQs. Transcripts were analyzed using the survey codebook and de novo codes to generate themes. We used the focus group findings to finalize the checklist.

RESULTS

Seventy-five students (82%) completed the survey. Students identified "structure" (organizing into sections/lists) and "content" (clearly delineating student's task) as two ways to improve SAQ quality. Eight faculty participated in focus groups of two to three participants each. Faculty expanded on these previous themes and also identified a new theme, "process." This included peer review of SAQs as well as the iterative process involved in writing the SAQ, model answer, and grading rubric.

CONCLUSIONS

Content, structure, and process were the three areas identified for the improvement of SAQ quality at our institution. A checklist outlining best practices in these areas may be best implemented and adopted within the SAQ peer-review process.

Online assessment in undergraduate medical education: Challenges and solutions from a LMIC university

BACKGROUND AND OBJECTIVES

The Covid-19 pandemic has caused large-scale disruption in almost all educational programs across the world. Planning and rapid implementation of assessment through an online format presents the next set of novel challenges that must be addressed by academic administrations across the globe.

METHODS

This cross-sectional study was conducted between March to August 2020 at the Aga Khan University Medical College. Two hundred medical students of year 1 and 2 participated in the study. We describe the planning, processes, and outcomes of online assessments using video communication platforms conducted at a private university in Pakistan. Standardized protocols were written and piloted, extensive training of student, proctors and staff for preparation and conduct of online assessments were developed. Feedback was recorded after each session and suggestions were incorporated in subsequent high-stakes assessments.

RESULTS

A total of three pilot assessments were conducted to identify issues and process refinement. Commercially available lockdown browser and ZOOM were used in the first pilot; 80% of the class was unable to launch lockdown browser and laptops required repeated reload/reboot. For the second pilot assessment, University's VLE page & MS Teams was trailed. Issues with internet connectivity, VLE page slowdown, and suboptimal recording feature in MS Teams were identified. For the final pilot assessment, phased launching of VLE page with single test item per page was implemented with success. The students reported that attempting the online exam on VLE with ZOOM support was user friendly. Ninety percent of the class was supportive of the continuing with the online assessments.

CONCLUSION

In order to device an effective protocol for e-assessments conducting multiple trial runs, and incorporating feedback from all stakeholders is a necessity.

Perceptions of medical students in Pakistan, KSA, and the US regarding the significance of case-based learning

Objective

This study aims to determine the perceptions of medical students in Pakistan, KSA, and the US regarding the significance of case-based learning (CBL).

Methods

For this cross-sectional study, data were collected by administering an online questionnaire to students in medical schools across Pakistan, KSA, and the US.

Results

A total of 344 medical students participated in this study, the great majority of whom agree that CBL paves the way for developing a sound understanding of the core subject, provides insight into real-life experiences, helps them transform from fact memorisers into problem solvers, and keeps them engaged during sessions, which motivates them to attend more of these. A comparison of respondents from Pakistan and KSA shows that CBL promotes deep learning and fostered their critical thinking; however, there was a difference in perception in some categories, including CBL as a tool used for grasping key concepts (p = 0.004), providing insight into real-life experiences (p = 0.001), offering a platform for self-directed learning (p = 0.000), nurturing collaborative abilities (p = 0.004), and maintaining students’ engagement (p = 0.002).

Conclusion

Our study shows that the selected cohort of medical students perceive CBL as an effective learning tool, as the majority feel overwhelmingly positive towards it. This study thus proposes the introduction of clinical exposure for medical students early in MBBS programmes, which will help promote collaborative skills and self-directed learning among them.

Faculty development program assists the new faculty in constructing high-quality short answer questions; a quasi-experimental study

Faculty development programs (FD) prepare the faculty for their educational role and career tasks. We aimed to evaluate the effectiveness of FDP in advancing the quality of short-answer questions (SAQs). This was a quasi-experimental study, comprising 37 new faculty. The SAQs were examined on psychometric analysis and Bloom’s cognitive levels for the two educational blocks of 1st medical year (i.e. Musculoskeletal (MSK) and Renal blocks). We found substantial improvement in the discrimination index values of SAQs prepared after the workshop (p = 0.04). A higher number of SAQs with moderate difficulty and higher discrimination were also observed. Flaws within the post-workshop questions were reduced (3.0%) when compared with pre-workshop (12.5%). The major incline was also reported within Bloom’s cognitive levels when pre-workshop K2 questions (30%) were compared with post-workshop (45.5%) with a p-value = 0.05. The SAQs constructed by the faculty member without participating in FDP are generally of unsatisfactory quality. After the FDP the assessment items of two blocks improved for various parameters of student assessment. The current study advocates that newly joined faculty shall be provided with the FDP to be guided, trained and supported for improving the quality of assessment through SAQs items writing.

Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions

OBJECTIVES

To analyze the low to medium distractor efficiency items in a multiple-choice question (MCQ) paper for item writing flaws.

METHODS

This qualitative study was conducted at Islamic International Medical College Rawalpindi, in October 2019. Archived item- analysis report from a midyear medium stakes MCQ paper of 2nd year MBBS class, was analyzed to determine the non-functional distractors (NFDs) and distractor efficiency (DE) of items, in a total of 181 MCQs. DE was categorized as low (3-4 NFDs), medium (1-2 NFDs) and high (0 NFD). Subsequently, qualitative document analysis of the MCQ paper whose item analysis report was assessed was conducted to investigate the item flaws in the low to medium DE items. The flaws identified were coded and grouped as, within option flaws, alignment flaws between options and stem/ lead-in and other flaws.

RESULTS

Distractor efficiency was high in 69 items (38%), moderate in 75 items (42%) and low in 37 items (20%). The item-writing flaws identified in low to moderate DE items within distractors included, non-homogenous length (1.8%), non-homogenous content (8%) and repeat in distractor (1.7%). Alignment flaws between distractors and stem/ lead-in identified were linguistic cues (10%), logic cues (12.5%) and irrelevant distractors (16%). Flaws unrelated to distractors were low cognitive level items (40%) and unnecessarily complicated stems (11.6%).

CONCLUSIONS

Analyzing the low to medium DE items for item writing flaws, provides valuable information about item writing errors which negatively impact the distractor efficiency.

Development of a New Scoring System To Accurately Estimate Learning Outcome Achievements via Single, Best-Answer, Multiple-Choice Questions for Preclinical Students in a Medical Microbiology Course

During the preclinical years, single-best-answer multiple-choice questions (SBA-MCQs) are often used to test the higher-order cognitive processes of medical students (such as application and analysis) while simultaneously assessing lower-order processes (like knowledge and comprehension). Consequently, it can be difficult to pinpoint which learning outcome has been achieved or needs improvement. We developed a new scoring system for SBA-MCQs using a step-by-step methodology to evaluate each learning outcome independently. Enrolled in this study were third-year medical students (n = 316) who had registered in the basic microbiology course at the Faculty of Medicine, Siriraj Hospital, Mahidol University during the academic year 2017. A step-by-step SBA-MCQ with a new scoring system was created and used as a tool to evaluate the validity of the traditional SBA-MCQs that assess two separate outcomes simultaneously. The scores for the two methods, in percentages, were compared using two different questions (SBA-MCQ1 and SBA-MCQ2). SBA-MCQ1 tested the students' knowledge of the causative agent of a specific infectious disease and the basic characteristics of the microorganism, while SBA-MCQ2 tested their knowledge of the causative agent of a specific infectious disease and the pathogenic mechanism of the microorganism. The mean score obtained with the traditional SBA-MCQs was significantly lower than that obtained with the step-by-step SBA-MCQs (85.9% for the traditional approach versus 90.9% for step-by-step SBA-MCQ1; p < 0.001; and 81.5% for the traditional system versus 87.4% for step-by-step SBA-MCQ2; p < 0.001). Moreover, 65.8% and 87.8% of the students scored lower with the traditional SBA-MCQ1 and the traditional SBA-MCQ2, respectively, than with the corresponding sets of step-by-step SBA-MCQ questions. These results suggest that traditional SBA-MCQ scores need to be interpreted with caution because they have the potential to underestimate the learning achievement of students. Therefore, the step-by-step SBA-MCQ is preferable to the traditional SBA-MCQs and is recommended for use in examinations during the preclinical years.

Evaluating the Quality of Multiple Choice Question in Paediatric Dentistry Postgraduate Examinations

Objectives

This study aimed to evaluate the quality of multiple choice question (MCQ) items in two postgraduate paediatric dentistry (PD) examinations by determining item writing flaws (IWFs), difficulty index (DI) and cognitive level.

Methods

This study was conducted at Mohamed Bin Rashid University of Medicine and Health Sciences, Dubai, UAE. Virtual platform-based summative versions of the general paediatric medicine (GPM) and prevention of oral diseases (POD) examinations administered during the second semester of the 2017–2018 academic year were used. Two PD faculty members independently reviewed each question to assess IWFs, DI and cognitive level.

Results

A total of 185 single best answer MCQs with 4–5 options were analysed. Most of the questions (81%) required information recall, with the remainder (19%) requiring higher levels of thinking and data explanation. The most common errors among IWFs were the use of “except” or “not” in the lead-in, tricky or unfocussed stems and opportunities for students to use convergence strategies. There were more IWFs in the GPM than the POD examination, but this was not statistically significant (P = 0.105). The MCQs in the GPM and POD examination were considered easy since the mean DIs (89.1% ± 8.9% and 76.5% ± 7.9%, respectively) were more than 70%.

Conclusion

Training is an essential element of adequate MCQ writing. A general comprehensive review of all programme’s MCQs is needed to emphasise the importance of avoiding IWFs. A faculty development programme is recommended to improve question-writing skills in order to align examinations with programme learning outcomes and enhance the ability to measure student competency through questions requiring higher level thinking.

Medical students' preferences towards learning resources and their study habits at King Abdulaziz University, Jeddah, Saudi Arabia

OBJECTIVE

The purpose of the present study was to explore why our medical students are avoiding the study of professional textbooks. We conducted this study from 10th March to 15th May 2017 at the King Abdulaziz University (KAU), Jeddah, to investigate their preferences towards learning resources, their study habits and correlation of academic achievements as a result of these trends. A questionnaire was provided as a web link. The participants of the study included medical students and data was analyzed on SPSS-Version 21.

RESULTS

A total of 347/500 medical students participated in the study. Among our participants, there were 123 (35.5%) males and 224 (64.6%) were females. Female students' spent most of their time reading textbooks as compared to males (P-value = 0.001). Males mostly preferred the lecture handouts provided by their teachers. One-third of students admitted that, due to lack of a good grasp of English, they do not readily understand textbooks and consequently 67 (19.3%) students' showed a lack of interest in textbooks. Majority of the males 103 (84%) spent twice a time (66 vs. 33%) watching television as compared to the females. WhatsApp and Facebook kept both the sexes busy in most of their spare time.