Using prescribing very short answer questions to identify sources of medication errors: a prospective study in two UK medical schools

Comparison of Online Peer-Assisted Learning and Faculty-Led Teaching for Short Answer Questions

INTRODUCTION

Peer-assisted learning (PAL) is a well-recognised pedagogical approach in medical education; however, research on its effectiveness in online settings remains limited. Multiple-choice questions have been the predominant method for assessing PAL outcomes, despite Short Answer Questions (SAQs) being the superior tool for evaluating knowledge. This study compares online peer and faculty teaching in enhancing medical students' higher-order thinking skills and assesses students' perceptions of these methods.

METHODS

Third-year medical students undergoing surgical placements were consented and recruited for the study. Three pre-defined cohorts were randomised to the following arms: no intervention (n = 41), online PAL teaching (n = 37) and online faculty teaching (n = 35). Peer teaching was delivered by fourth-year students (n = 6) and faculty teaching by Clinical Teaching Fellows (CTFs) (n = 6). Academic outcomes were assessed using end-of-block SAQ formatives, and teaching quality was evaluated using the validated SEEQ questionnaire. Knowledge gain and self-perceived confidence were assessed through pre- and post-session tests, validated with a reference group of learners.

RESULTS

Consent for SAQ exam scores was obtained from n = 19 (no intervention), n = 29 (PAL) and n = 21 (CTF). No significant differences were seen between the groups (p = 0.650). SEEQ completion was n = 24 (PAL) and n = 30 (CTF). CTF tutors received significantly higher ratings in domains of Learning (p = 0.017) and Group Interaction (p = 0.036). Pre- and post-session tests showed no significant differences in scores (p = 0.957) or self-perceived confidence ratings (p = 0.454).

CONCLUSION

This study shows that online PAL is a viable alternative to faculty-led teaching for enhancing SAQ skills and knowledge acquisition. However, faculty-led teaching offers a superior educational experience.

Development of Immediate Self-Feedback Very Short Answer Questions: Implementing Testlet Response Theory in Formative Examinations Across Multiple Occasions

Background

The novel Immediate Self-feedback Very Short Answer Question (ISF-VSAQ) format exhibited testlet effect due to its instant feedback feature. This study utilized the testlet-as-a-polytomous-item-modeling (TPIM), an extension of item response theory, to tackle the testlet-related challenges.

Methods

Ninety-four pre-clinical students participated in two ten-item ISF-VSAQ exams on cardiovascular drugs based on a parallel blueprint. Each question involves clinical vignettes followed by expected answers, self-marking, and understanding levels. Messick's validity framework guided the collection of validity evidence.

Results

Validity evidence included five sources: (1) Content: Experts reviews and alignment with the standard examination blueprint. (2) Response process: Students received an example and guide of ISF-VSAQ. Self-assessment steps were also explained, and all students were provided adequate time. (3) Internal structure: Higher marginal reliability was demonstrated in the TPIM model. While unidimensionality was violated in the second ISF-VSAQ, the TPIM maintained unidimensionality. Item discrimination was generally acceptable, except for one item on the second occasion with minor vignette changes. Items with new knowledge showed better discrimination and item information functions on the second occasion. (4) Relations to other variables: Path analysis shows that the first occasion's understanding and scoring abilities strongly influenced the second occasion, highlighting the testlet effect. (5) Consequences: Path analysis revealed that the understanding ability in the second attempt directly affected the summative cardiovascular (β = 0.26, p = 0.066) and comprehensive examination ability (β = 0.34, p = 0.010).

Conclusion

The TPIM model effectively tackles the testlet effect of the present ISF-VSAQ. Thus, when employing multiple instances of self-feedback format, analyzing it as a testlet is recommended.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-024-02167-w.

Development and validation of immediate self-feedback very short answer questions for medical students: practical implementation of generalizability theory to estimate reliability in formative examination designs

BACKGROUND

Very Short Answer Questions (VSAQs) reduce cueing and simulate better real-clinical practice compared with multiple-choice questions (MCQs). While integrating them into formative exams has potential, addressing marking time and ideal occasions and items is crucial. This study gathers validity evidence of novel immediate self-feedback VSAQ (ISF-VSAQ) format and determines the optimal number of items and occasions for reliable assessment.

METHODS

Ninety-four third-year pre-clinical students took two ten-item ISF-VSAQ exams on cardiovascular drugs. Each question comprised two sections: (1) Questions with space for student responses and (2) a list of possible correct answers offering partial-credit scores ranging from 0.00 to 1.00, along with self-marking and self-feedback options to indicate whether they fully, partially, or did not understand the possible answers. Messick's validity framework guided the collection of validity evidence.

RESULTS

Validity evidence included five sources: (1) Content: The expert reviewed the ISF-VSAQ format, and the question was aligned with a standard examination blueprint. (2) Response process: Before starting, students received an example and guide to the ISF-VSAQ, and the teacher detailed the steps in the initial session to aid self-assessment. Unexpected answers were comprehensively reviewed by experts. (3) Internal structure: The Cronbach alphas are good for both occasions (≥ 0.70). A generalizability study revealed Phi-coefficients of 0.60, 0.71, 0.76, and 0.79 for one to four occasions with ten items, respectively. One occasion requires twenty-five items for acceptable reliability (Phi-coefficient = 0.72). (4) Relations to other variables: Inter-rater reliability between self-marking and teacher is excellent for each item (rs(186) = 0.87-0.98,p = 0.001). (5) Consequences: Path analysis revealed that the self-reflected understanding score in the second attempt directly affected the final MCQ score (β = 0.25,p = 0.033). However, the VSAQ score did not. Regarding perceptions, over 80% of students strongly agreed/agreed that the ISF-VSAQ format enhances problem analysis, presents realistic scenarios, develops knowledge, offers feedback, and supports electronic usability.

CONCLUSION

Electronic ISF-VSAQs enhanced understanding elevates learning outcomes, rendering them suitable for formative assessments with clinical scenarios. Increasing the number of occasions effectively enhances reliability. While self-marking is reliable and may reduce grading efforts, instructors should review answers to identify common student errors.

Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

Effectiveness of short, personalised student assistantships: an evaluative study across eight London hospitals

OBJECTIVES

Student assistantships are recommended to prepare medical graduates for clinical practice. Traditionally, assistantships have consisted of longer placements, often up to 15 weeks. However, within the constraints of the final year, medical schools need to carefully balance the time required for specialty placements, assessments and the risk of student burnout. We set out to evaluate the effectiveness of shorter, personalised student assistantships.

DESIGN

An evaluative study on the changes in final year student confidence in preparedness for practice after a 3-week assistantship with defined learning objectives and learning needs assessment.

SETTING

Eight hospitals affiliated with Imperial College School of Medicine.

OUTCOMES

Student confidence in 10 learning outcomes including organising ward rounds, documentation, communication with colleagues, communication with patients and relatives, patient handover, practical procedures, patient management, acute care, prioritisation and out-of-hours clinical work.

RESULTS

Two hundred and twenty final year medical students took part in the student assistantship, of whom 208 completed both the pre-assistantship and post-assistantship confidence rating questionnaires (95% completion rate). After the assistantship, 169 (81%) students expressed increased confidence levels in one or more learning objectives. For each individual learning objective, there was a significant change in the proportion of students who agreed or strongly agreed after the assistantship (p<0.0001).

CONCLUSION

Overall, the focused 3-week, personalised student assistantships led to significant improvement across all learning objectives related to preparedness for practice. The use of the pre-assistantship confidence rating questionnaire allowed students to identify and target areas of learning needs during their assistantship.

Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?

BACKGROUND

We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations.

METHODS

Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set.

RESULTS

A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z = -2.82, p = 0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z = 2.46, p = 0.014).

CONCLUSIONS

When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.

Transforming MRCPsych theory examinations: digitisation and very short answer questions (VSAQs)

Many examinations are now delivered online using digital formats, the migration to which has been accelerated by the COVID-19 pandemic. The MRCPsych theory examinations have been delivered in this way since Autumn 2020. The multiple choice question formats currently in use are highly reliable, but other formats enabled by the digital platform, such as very short answer questions (VSAQs), may promote deeper learning. Trainees often ask for a focus on core knowledge, and the absence of cueing with VSAQs could help achieve this. This paper describes the background and evidence base for VSAQs, and how they might be introduced. Any new question formats would be thoroughly piloted before appearing in the examinations and are likely to have a phased introduction alongside existing formats.

Thinking differently - Students' cognitive processes when answering two different formats of written question

BACKGROUND

Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students' cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions.

METHODS

We undertook a 'think aloud' study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used.

RESULTS

Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students.

CONCLUSIONS

Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.

Exploring the feasibility of using very short answer questions (VSAQs) in team-based learning (TBL)

BACKGROUND

Team-based learning (TBL) currently relies on single best answer questions (SBAQs) to provide immediate feedback. Very short answer questions (VSAQs) are a reliable and discriminatory alternative that encourage learners to use more authentic clinical reasoning strategies compared to SBAQs. However, the challenge of marking VSAQs has limited their integration into TBL; we therefore explored the feasibility of VSAQs within a TBL session.

METHODS

An online platform was developed to allow immediate marking of VSAQs during the TBL sessions. As part of the readiness assurance process, students completed VSAQs and SBAQs, which were marked in real time.

RESULTS

Instructors were able to mark all VSAQs during the individual readiness assurance test (iRAT), which facilitated the provision of immediate feedback during the team readiness assurance test (tRAT). The mean time to mark five VSAQs was 422 seconds (SD 73 seconds). For VSAQs, the number of attempts to reach the correct answer ranged from 1 to 38, compared to 1 to 4 for SBAQs. In total, 71.6% of students agreed that using VSAQs in TBL helped to emphasise group discussions.

DISCUSSION

The wide range of attempts at, and students' perspectives of VSAQs are suggestive of their positive impact on student discussion during TBL. We demonstrate how new technology allows VSAQs to be feasibly integrated into TBL with the potential to enrich group discussions.

Twelve tips for facilitating medical students prescribing learning on clinical placement

Prescribing is a complex clinical skill requiring mastery by the end of basic medical training. Prescribing errors are common in newly qualified doctors, aligned with expressed anxiety about prescribing, particularly with high-risk medications. Learning about prescribing needs to start early in medical training, underpinned by regular opportunities for reflective practice. Authentic learning within the clinical work environment is more effective than lecture based learning and allows potential immediate feedback. Educational strategies should support prescribing learning underpinned by appropriate formative and summative assessments. Students should routinely be expected to use resources including an online formulary, sustained through tracking individual progress through use of their own personal formulary or 'p' drugs. Regular prescribing practice with embedded feedback during undergraduate training will help to ensure newly qualified doctors are more confident and competent prescribers.