Best Practices Related to Examination Item Construction and Post-hoc Review

What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study

PURPOSE

The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework.

METHODS

This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs.

RESULTS

A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format).

CONCLUSIONS

CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions

PROBLEM

Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education.

INTERVENTION

Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability.

CONTEXT

This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school.

IMPACT

Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric).

LESSONS LEARNED

Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

Teaching LGBTQ+ Health, a Web-Based Faculty Development Course: Program Evaluation Study Using the RE-AIM Framework

BACKGROUND

Many health professions faculty members lack training on fundamental lesbian, gay, bisexual, transgender, and queer (LGBTQ+) health topics. Faculty development is needed to address knowledge gaps, improve teaching, and prepare students to competently care for the growing LGBTQ+ population.

OBJECTIVE

We conducted a program evaluation of the massive open online course Teaching LGBTQ+ Health: A Faculty Development Course for Health Professions Educators from the Stanford School of Medicine. Our goal was to understand participant demographics, impact, and ongoing maintenance needs to inform decisions about updating the course.

METHODS

We evaluated the course for the period from March 27, 2021, to February 24, 2023, guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. We assessed impact using participation numbers, evidence of learning, and likelihood of practice change. Data included participant demographics, performance on a pre- and postcourse quiz, open-text entries throughout the course, continuing medical education (CME) credits awarded, and CME course evaluations. We analyzed demographics using descriptive statistics and pre- and postcourse quiz scores using a paired 2-tailed t test. We conducted a qualitative thematic analysis of open-text responses to prompts within the course and CME evaluation questions.

RESULTS

Results were reported using the 5 framework domains. Regarding Reach, 1782 learners participated in the course, and 1516 (85.07%) accessed it through a main course website. Of the different types of participants, most were physicians (423/1516, 27.9%) and from outside the sponsoring institution and target audience (1452/1516, 95.78%). Regarding Effectiveness, the median change in test scores for the 38.1% (679/1782) of participants who completed both the pre- and postcourse tests was 3 out of 10 points, or a 30% improvement (P<.001). Themes identified from CME evaluations included LGBTQ+ health as a distinct domain, inclusivity in practices, and teaching LGBTQ+ health strategies. A minority of participants (237/1782, 13.3%) earned CME credits. Regarding Adoption, themes identified among responses to prompts in the course included LGBTQ+ health concepts and instructional strategies. Most participants strongly agreed with numerous positive statements about the course content, presentation, and likelihood of practice change. Regarding Implementation, the course cost US $57,000 to build and was intramurally funded through grants and subsidies. The course faculty spent an estimated 600 hours on the project, and educational technologists spent another 712 hours. Regarding Maintenance, much of the course is evergreen, and ongoing oversight and quality assurance require minimal faculty time. New content will likely include modules on transgender health and gender-affirming care.

CONCLUSIONS

Teaching LGBTQ+ Health improved participants' knowledge of fundamental queer health topics. Overall participation has been modest to date. Most participants indicated an intention to change clinical or teaching practices. Maintenance costs are minimal. The web-based course will continue to be offered, and new content will likely be added.

Information overdose: Student performance and perceptions of pharmacology resources on exams

Physicians of the future will be expected to synthesize new knowledge and appropriately apply it in patient care. Here, we report on the effects of and student attitudes towards resource-enhanced exams by comparing student performance on closed-book exams with or without access to pharmacology flashcards. Setting: the University of California, San Francisco (UCSF) School of Medicine (SOM), class of 2021 (N = 149), followed over 4 years. We provided pharmacology flashcards for studying purposes in all blocks; flashcards were only accessible during closed-book exams in 2 of 5 blocks. We collected pharmacology open-ended question (OEQ) scores and analyzed results using repeated measures ANOVA (SPSS). We collected MS4 survey data using Qualtrics and conducted a thematic content analysis. Performance on pharmacology questions on exams was not higher with access to pharmacology flashcards during exams. The number of students who passed pharmacology questions without flashcards on exams was as follows: 137 ± 3.7, 132 ± 5.0, and 134 ± 7.9 (average ± SEM). The number of students who passed pharmacology questions with flashcards on exams was as follows: 132 ± 6.6 and 120 ± 7.5. Survey comments revealed several themes. Access to pharmacology flashcards during exams allowed learners to focus on understanding the bigger picture and reduced stress. A subset of students reported having access to flashcards on pre-clerkship exams hurt their preparation for clerkships. Flashcards as exam resources were received well by approximately half the class, who reported benefits including more time to focus on understanding bigger picture concepts and reduced stress.

Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure

Objective. To determine whether there are differences in the performance and quality of multiple-choice items with opposite stem orientations (positive or negative), based on a novel item quality measure and conventional psychometric parameters.Methods. A retrospective study was conducted on multiple-choice assessment items used in years two and three of pharmacy school for pharmacotherapy and related courses administered between August 2018 and December 2019. Conventional psychometric parameters (difficulty and discrimination indices), average response time, nonfunctional distractor percentage, and a novel measure of item quality of negatively worded items were compared with those of control items, namely positively worded items (n=103 each). This novel measure uses difficulty and discrimination in tandem for the decision to reject, review, or retain items in an assessment. Statistical analyses were performed on continuous and categorical variables, on the relationship between difficulty and discrimination, and on differences in correlation coefficients between positively and negatively worded items.Results. Stem orientation was not significantly associated with the novel measure of item quality. Also, there were no significant differences between positively and negatively worded items in any of the psychometric parameters. There were significant, negative correlations between difficulty and discrimination indices in both groups, and the correlation coefficients were significantly stronger in positively versus negatively worded items.Conclusion. Items with opposite stem orientations show no differences in the novel item quality measure nor in conventional measures of performance and quality, except in difficulty-discrimination relationships. This suggests that negatively worded items should be used when necessary, but cautiously.

The effect of access to electronic resources during examination on medical and dental students scores in summative assessment: quasi experimental study

BACKGROUND

Access to electronic (E) resources has become an indispensable requirement in medical education and practice.

OBJECTIVE

Our objective was to assess the effect of E-resources access during examination on end-course-exam scores of medical and dental students.

METHODS

A quasi-experimental study which included two cohorts of medical (n = 106 & 85) and three cohorts of dental students (n = 66, 64 and 69) who took end-course- exams. Each exam was composed of two- parts (Part I and II), that encompassed equal number of questions and duration. Access to E-resources was allowed in part-II only. Items Difficulty Index (DI), Discrimination Index, (DisI), Point Biserial, (PBS) and cognitive level were determined.

RESULTS

The study included 390 students. The proportion of items at various levels of DI, DisI, and PBS and the average values for item DI, DisI in both parts of each exam were comparable. The average scores in part-II were significantly higher than part-I (P < 0.001, < 0.001 and 0.04) and lower-order cognitive-level items scores were higher in three exams (P < 0.0001, 0.0001, 0.0001). Higher- order cognitive level items scores were comparable between part I and II in all courses. The significant factor for change in marks were questions cognitive level and type of the course.

CONCLUSION

Access to E-resources during examination does not make a significant difference in scores of higher-order cognitive level items. Question cognitive level and course type were the significant factors for the change in exam scores when accessing E-resources. Time-restricted E-resources accessed tests that examine higher cognitive level item had no significant academic integrity drawback.

Focusing on cultural humility in pharmacy assessment tools

INTRODUCTION

Many pharmacy programs have been reviewing their curricula to ensure that unconscious bias is not interwoven throughout. However, few programs have looked at their assessment tools to determine if unconscious bias exists in them. Since research on unconscious bias in assessment tools is currently limited, this study was developed to explore how race, ethnicity, age, and gender were used in multiple-choice item banks at one pharmacy program.

METHODS

A total of 27 pharmacy test banks, including examinations from first-, second-, and third-year pharmacy curricula, were examined. The exams administered in the 27 courses provided 3621 questions for review and coding based on reference to race, ethnicity, age, and gender.

RESULTS

Of the total 3621 exam items reviewed, 40 referenced race. Of those 40, it was determined that race was only applicable/relevant to two (5%) of the questions, linked to Whites. Gender breakdown was 56% male and 45% female, with no mention of transgender. Gender was central to 72 of the 426 questions. Age was mentioned in 381 of the questions and was central in only 46. Ethnicity was not mentioned in any of the questions. In none of the questions were these variables self-identified.

CONCLUSIONS

The routine mention of race in preparation of materials such as question banks, with no specified goal, risks contributing to racial bias. The goal of all pharmacy educators is to graduate pharmacists that possess sound academic knowledge, clinical skills, and the cultural humility to provide quality care to their patients.

Should multiple-choice questions get the SAQ? Development of a short-answer question writing rubric

INTRODUCTION

Short-answer questions (SAQs) are often used to assess pharmacy student competency. However, the literature lacks guidance on SAQ development strategies, resulting in varying practices between SAQ writers. Understanding student and faculty perceptions of what constitutes a high-quality SAQ can identify best practices for SAQ development.

METHODS

We surveyed second-year pharmacy students at the University of California San Francisco (UCSF) to assess their perceptions of SAQs. Likert-type data were descriptively analyzed, and open-ended responses were analyzed using thematic analysis; we used these results to draft an initial SAQ checklist. We then conducted focus groups of UCSF pharmacy faculty to explore their experiences writing SAQs. Transcripts were analyzed using the survey codebook and de novo codes to generate themes. We used the focus group findings to finalize the checklist.

RESULTS

Seventy-five students (82%) completed the survey. Students identified "structure" (organizing into sections/lists) and "content" (clearly delineating student's task) as two ways to improve SAQ quality. Eight faculty participated in focus groups of two to three participants each. Faculty expanded on these previous themes and also identified a new theme, "process." This included peer review of SAQs as well as the iterative process involved in writing the SAQ, model answer, and grading rubric.

CONCLUSIONS

Content, structure, and process were the three areas identified for the improvement of SAQ quality at our institution. A checklist outlining best practices in these areas may be best implemented and adopted within the SAQ peer-review process.

The Health Improvement Profile for people with severe mental illness: Feasibility of a secondary analysis to make international comparisons

WHAT IS ALREADY KNOWN ABOUT THIS TOPIC?: Individuals with severe mental illness (SMI) have elevated risks for physical health problems and low screening rates. No previous studies have compared the physical health promotion needs of people with SMI using the same screening tool across different international settings. WHAT THIS PAPER ADDS TO EXISTING KNOWLEDGE?: It appears feasible to use the HIP to profile and compare physical health-related risks in people with SMI across different international settings. The HIP tool identified significant differences in areas of risk across the four countries. WHAT ARE THE IMPLICATIONS FOR PRACTICE?: The HIP could be used to identify unique clusters of health promotion needs in different countries. Use of HIP health checks may support implementation of individualized interventions. ABSTRACT: Introduction To date, no studies have contrasted physical health profiles of people with severe mental illness (SMI) in different countries. Aim To evaluate feasibility of using the Health Improvement Profile (HIP) to compare and contrast physical health and health behaviours of people with SMI from four countries. Method An observational feasibility study using secondary analysis of pooled health state and lifestyle data. Physical health checks using modified versions of HIP were administered in four countries. Results Findings suggest feasibility of HIP screening to profile and compare physical health and health behaviours of people with SMI across international settings. High overall numbers of risk items (red flags) were identified in all but the Thailand sample. Despite some commonalities, there were important differences in health profiles across countries. Discussion This is the first study to demonstrate feasibility of the HIP to compare health risks in individuals with SMI across countries. Future multi-national HIP studies should recruit a fully powered stratified random sample of people with SMI that is representative of each setting. Implications for practice It appears feasible to utilize the HIP to identify specific areas of health risk in different countries, which may help to better focus nursing interventions and use of resources.

Multiple-Choice Tests: A-Z in Best Writing Practices

Multiple-choice tests are the most used method of assessment in medical education. However, there is limited literature in medical education and psychiatry to inform the best practices in writing good-quality multiple-choice questions. Moreover, few physicians and psychiatrists have received training and have experience in writing them. This article highlights the strategies in writing high-quality multiple-choice items and discusses some common flaws that can impact validity and reliability of the assessment examinations.

Getting Rigor Right

A colleague of mine once shared a story he heard of an organic chemistry teacher who would tell people he had one of the most important, lifesaving jobs in the world. Why? Because he made sure academically unprepared premed students never made it to medical school. Teachers can have many possible motivations for maintaining high academic standards in their courses. A desire to ensure future health practitioners are adequately prepared to engage in the intellectual rigor of their profession may be just one of them.