Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study

Development and validation of immediate self-feedback very short answer questions for medical students: practical implementation of generalizability theory to estimate reliability in formative examination designs

BACKGROUND

Very Short Answer Questions (VSAQs) reduce cueing and simulate better real-clinical practice compared with multiple-choice questions (MCQs). While integrating them into formative exams has potential, addressing marking time and ideal occasions and items is crucial. This study gathers validity evidence of novel immediate self-feedback VSAQ (ISF-VSAQ) format and determines the optimal number of items and occasions for reliable assessment.

METHODS

Ninety-four third-year pre-clinical students took two ten-item ISF-VSAQ exams on cardiovascular drugs. Each question comprised two sections: (1) Questions with space for student responses and (2) a list of possible correct answers offering partial-credit scores ranging from 0.00 to 1.00, along with self-marking and self-feedback options to indicate whether they fully, partially, or did not understand the possible answers. Messick's validity framework guided the collection of validity evidence.

RESULTS

Validity evidence included five sources: (1) Content: The expert reviewed the ISF-VSAQ format, and the question was aligned with a standard examination blueprint. (2) Response process: Before starting, students received an example and guide to the ISF-VSAQ, and the teacher detailed the steps in the initial session to aid self-assessment. Unexpected answers were comprehensively reviewed by experts. (3) Internal structure: The Cronbach alphas are good for both occasions (≥ 0.70). A generalizability study revealed Phi-coefficients of 0.60, 0.71, 0.76, and 0.79 for one to four occasions with ten items, respectively. One occasion requires twenty-five items for acceptable reliability (Phi-coefficient = 0.72). (4) Relations to other variables: Inter-rater reliability between self-marking and teacher is excellent for each item (rs(186) = 0.87-0.98,p = 0.001). (5) Consequences: Path analysis revealed that the self-reflected understanding score in the second attempt directly affected the final MCQ score (β = 0.25,p = 0.033). However, the VSAQ score did not. Regarding perceptions, over 80% of students strongly agreed/agreed that the ISF-VSAQ format enhances problem analysis, presents realistic scenarios, develops knowledge, offers feedback, and supports electronic usability.

CONCLUSION

Electronic ISF-VSAQs enhanced understanding elevates learning outcomes, rendering them suitable for formative assessments with clinical scenarios. Increasing the number of occasions effectively enhances reliability. While self-marking is reliable and may reduce grading efforts, instructors should review answers to identify common student errors.

Foresight-a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study

BACKGROUND

An electronic health record (EHR) holds detailed longitudinal information about a patient's health status and general clinical history, a large portion of which is stored as unstructured, free text. Existing approaches to model a patient's trajectory focus mostly on structured data and a subset of single-domain outcomes. This study aims to evaluate the effectiveness of Foresight, a generative transformer in temporal modelling of patient data, integrating both free text and structured formats, to predict a diverse array of future medical outcomes, such as disorders, substances (eg, to do with medicines, allergies, or poisonings), procedures, and findings (eg, relating to observations, judgements, or assessments).

METHODS

Foresight is a novel transformer-based pipeline that uses named entity recognition and linking tools to convert EHR document text into structured, coded concepts, followed by providing probabilistic forecasts for future medical events, such as disorders, substances, procedures, and findings. The Foresight pipeline has four main components: (1) CogStack (data retrieval and preprocessing); (2) the Medical Concept Annotation Toolkit (structuring of the free-text information from EHRs); (3) Foresight Core (deep-learning model for biomedical concept modelling); and (4) the Foresight web application. We processed the entire free-text portion from three different hospital datasets (King's College Hospital [KCH], South London and Maudsley [SLaM], and the US Medical Information Mart for Intensive Care III [MIMIC-III]), resulting in information from 811 336 patients and covering both physical and mental health institutions. We measured the performance of models using custom metrics derived from precision and recall.

FINDINGS

Foresight achieved a precision@10 (ie, of 10 forecasted candidates, at least one is correct) of 0·68 (SD 0·0027) for the KCH dataset, 0·76 (0·0032) for the SLaM dataset, and 0·88 (0·0018) for the MIMIC-III dataset, for forecasting the next new disorder in a patient timeline. Foresight also achieved a precision@10 value of 0·80 (0·0013) for the KCH dataset, 0·81 (0·0026) for the SLaM dataset, and 0·91 (0·0011) for the MIMIC-III dataset, for forecasting the next new biomedical concept. In addition, Foresight was validated on 34 synthetic patient timelines by five clinicians and achieved a relevancy of 33 (97% [95% CI 91-100]) of 34 for the top forecasted candidate disorder. As a generative model, Foresight can forecast follow-on biomedical concepts for as many steps as required.

INTERPRETATION

Foresight is a general-purpose model for biomedical concept modelling that can be used for real-world risk forecasting, virtual trials, and clinical research to study the progression of disorders, to simulate interventions and counterfactuals, and for educational purposes.

FUNDING

National Health Service Artificial Intelligence Laboratory, National Institute for Health and Care Research Biomedical Research Centre, and Health Data Research UK.

Assessment approaches in undergraduate health professions education: towards the development of feasible assessment approaches for low-resource settings

BACKGROUND

Feasible and effective assessment approaches to measuring competency in health sciences are vital in competency-based education. Educational programmes for health professions in low- and middle-income countries are increasingly adopting competency-based education as a strategy for training health professionals. Importantly, the organisation of assessments and assessment approaches must align with the available resources and still result in the fidelity of implementation. A review of existing assessment approaches, frameworks, models, and methods is essential for the development of feasible and effective assessment approaches in low-resource settings.

METHODS

Published literature was sourced from 13 electronic databases. The inclusion criteria were literature published in English between 2000 and 2022 about assessment approaches to measuring competency in health science professions. Specific data relating to the aims of each study, its location, population, research design, assessment approaches (including the outcome of implementing such approaches), frameworks, models, and methods were extracted from the included literature. The data were analysed through a multi-step process that integrated quantitative and qualitative approaches.

RESULTS

Many articles were from the United States and Australia and reported on the development of assessment models. Most of the articles included undergraduate medical or nursing students. A variety of models, theories, and frameworks were reported and included the Ideal model, Predictive Learning Assessment model, Amalgamated Student Assessment in Practice (ASAP) model, Leadership Outcome Assessment (LOA) model, Reporter-Interpreter-Manager-Educator (RIME) framework, the Quarter model, and the model which incorporates four assessment methods which are Triple Jump Test, Essay incorporating critical thinking questions, Multistation Integrated Practical Examination, and Multiple Choice Questions (TEMM) model. Additional models and frameworks that were used include the Entrustable Professional Activities framework, the System of Assessment framework, the Reporter-Interpreter-Manager-Educator (RIME) framework, the Clinical Reasoning framework (which is embedded in the Amalgamated Student Assessment in Practice (ASAP) model), Earl's Model of Learning, an assessment framework based on the Bayer-Fetzer Kalamazoo Consensus Statement, Bloom's taxonomy, the Canadian Medical Education Directions for Specialists (CanMEDS) Framework, the Accreditation Council for Graduate Medical Education (ACGME) framework, the Dreyfus Developmental Framework, and Miller's Pyramid.

CONCLUSION

An analysis of the assessment approaches, frameworks, models, and methods applied in health professions education lays the foundation for the development of feasible and effective assessment approaches in low-resource settings that integrate competency-based education.

TRIAL REGISTRATION

This study did not involve any clinical intervention. Therefore, trial registration was not required.

Understanding medical professionalism using express team-based learning; a qualitative case-based study

Medical Professionalism (MP) defined as values, behaviours and attitudes that promote professional relationships, public trust and patient safety is a vital competency in health profession education. MP has a distinctive uniqueness due to cultural, contextual, conceptual, and generational variations. There is no standard instructional strategy to probe the understanding of MP in a cohesive, structured, interactive manner. This study aimed to investigate undergraduate medical students' understanding of MP using express team-based learning (e-TBL) at both campuses of Royal College of Surgeons Ireland (RCSI). Using the key principles of a sociocultural theoretical lens in adult learning theory, we designed e-TBL as a context-learning-based educational strategy. We conducted three e-TBL sessions on cross-cultural communication and health disparities, a reflective report on clinical encounters, and professionalism in practice. We collected, collated, and analyzed the student experiences qualitatively using data gathered from team-based case discussions during e-TBL sessions. A dedicated working group developed very short-answer questions for the individual readiness assurance test (IRAT) and MP-based case scenarios for team discussions. In this adapted 4-step e-TBL session, pre-class material was administered, IRAT was undertaken, and team-based discussions were facilitated, followed by facilitator feedback. A qualitative inductive thematic analysis was performed, which generated subthemes and themes illustrated in excerpts. Our thematic analysis of data from 172 students (101 from Bahrain and 71 from Dublin) yielded four unique themes: incoming professional attitudes, transformative experiences, sociological understanding of professionalism, and new professional identity formation. This qualitative study provides a deeper understanding of medical students' perceptions of medical professionalism. The generated themes resonated with divergent and evolving elements of MP in an era of socioeconomic and cultural diversity, transformative experiences, and professional identity formation. The core elements of these themes can be integrated into the teaching of MP to prepare fit-to-practice future doctors.

The validity of Engagement and Feedback Assessments (EFAs): identifying students at risk of failing

BACKGROUND

Imperial College School of Medicine, London UK, introduced a new curriculum in 2019, with a focus on the GMC outcomes for graduates, and pedagogy best practice. The new curriculum included formative assessments, named engagement and feedback assessments (EFAs), to support learning, and attainment in the summative examinations. The aims of this study were to assess the validity of EFAs and to determine whether they have utility as a modified form of programmatic assessment to inform decision-making regarding possible interventions by measuring and analysing attendance at and performance in these formative events.

METHODS

Seven hundred and sixty-one students were included in the study and assessment results were included for academic years 2019/20 to 2020/21. Forty-one data points per student, (27 in Year 1 and 14 in Year 2) were used, to compare EFA scores with the summative performance. Attendance was monitored through engagement with the EFAs.

RESULTS

Cohort 1 (enrolled 2019): In year 1, EFAs were associated with summative exam scores (overall r = 0.63, p < 0.001). Year 2, EFA scores were also associated with summative scores (overall r = 0.57, p < 0.001), including the clinical practical assessment (r = 0.45, p < 0.001). Missing two or more EFAs was associated with a significant increase in the likelihood of failing one or more summative examinations in the first year (OR: 7.97, 95% CI 2.65-34.39) and second year (OR: 3.20, 95% CI 1.74-5.95). Missing more than two EFAs in their first year was also associated with a higher risk of failing a summative examination in the second year (OR: 2.47, 95% CI 1.33-4.71). Students who increased their attendance between year 1 and 2 fared better in summative assessment than those who maintained poor attendance, whereas those that reduced their attendance fared worse than those that maintained high attendance. Cohort 2 (enrolled 2020): Analysis of cohort 2 supported these findings and in this cohort missing two or more EFAs was again associated with an increased likelihood of failing a summative examination (OR = 4.00, 95% CI = 2.02-7.90).

CONCLUSION

Our EFA model has validity in predicting performance in summative assessments and can inform prospective interventions to support students' learning. Enhancing attendance and engagement can improve outcomes.

Longitudinal assessment of undergraduate dental students: Building evidence for validity

PURPOSE

To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations.

METHODS

Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha.

RESULTS

Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included.

CONCLUSIONS

There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.

Tool to assess recognition and understanding of elements in Summary of Findings Table for health evidence synthesis: a cross-sectional study

of Findings (SoF) tables concisely present the main findings of evidence synthesis of health evidence, but how users navigate it to understand and interpret the presented information is not clear. We quantified the interaction of medical students with an SoF table while answering a knowledge quiz. Read&Learn tool was used to measure the number of target and non-target table cells visited for each question and the time spent on these cells. Students positively identified target elements for quiz questions and answered simpler questions, but struggled with critical thinking and understanding study outcomes. The question on outcomes with the largest improvement post-intervention had the fewest correct answers, the longest interaction with table cells and the most opened cells before answering. Students spent a median of 72% of the time reading target table cells. A heatmap of the interactions showed that they were mostly answer-oriented. Further development of the tool and metrics is needed to use the tool and the metrics to study the cognitive processes during the assessment of health evidence.

Assessing ChatGPT's ability to pass the FRCS orthopaedic part A exam: A critical analysis

AI technology has made significant advancements in recent years, with the notable development of ChatGPT in November 2022. Users have observed evidence of deductive reasoning, logical thinking, and coherent thought in ChatGPT's responses. This study aimed to determine if ChatGPT has the capability to pass the Orthopaedic Fellow of the Royal College of Surgeons (FRCS Orth) Part A exam.

METHODS

To assess ChatGPT4's ability to pass the Orthopaedic FRCS Orth Part A exam, a study was conducted using 240 mock FRCS Orth Part A questions. The study evaluated the accuracy of ChatGPT's answers and the response time for each question. Descriptive statistics were employed to analyse the chatbot's performance.

RESULTS

The evaluation revealed that ChatGPT4 achieved an overall score of 67.5% on Part A of the exam. However, ChatGPT4 did not meet the overall pass mark required for the FRCS Orth Part A exam.

CONCLUSION

This study demonstrates that ChatGPT was unable to pass the FRCS Orthopaedic examination. Several factors contributed to this outcome, including the lack of critical or high-order thinking abilities, limited clinical expertise, and the inability to meet the rigorous requirements of the exam.

A successful intervention to improve conceptual knowledge of medical students who rely on memorization of disclosed items

Background: The mere memorization of isolated facts without the claim of integrating them is detrimental to the desired learning outcomes in medical education. The current study was conducted within an environment where items from summative assessments are regularly disclosed by the university and consequently collected into an item bank that is shared between students. Within this environment, we aimed to quantify 1) to which extent students use disclosed items for their preparation for the summative exam, 2) a putative mismatch between (isolated factual) knowledge regarding distinct questions from the item bank and conceptual knowledge, and 3) to which extent this mismatch can be ameliorated by a project aiming to steer student learning away from the memorization of isolated facts toward the acquisition of conceptual knowledge. Methods: This steering project in the midst of the first semester consisted of the implementation of an oral exam based on selected learning objectives, preceded by two seminars. After their summative exam at the end of semester, 135 students performed a voluntary exam for study purposes. Here, authentic (i.e., presumably preknown) physiology questions taken from the item bank were used to assess students' ability to 1) recognize the correct answer in a multiple choice (MC) question, 2) recall the answer (short answer), or 3) display conceptual knowledge closely corresponding to the question presented in the other formats. Additionally, students received a questionnaire addressing their learning habits and attitudes. Results: The median reported percentage of learning time for the summative exam exclusively spent with this item bank was 80%. The results of the voluntary exam indicate that students frequently recognize and recall correct answers of included items without displaying knowledge of the underlying concept. Compared to recall of the correct answer, the probability of giving a correct answer regarding the corresponding basal physiologic concept was lower by 47 percentage points (p <0.001) for topics not included in the steering project. Regarding topics included in the steering project, this discrepancy was reduced to 25.5% (p <0.001). Conclusion: The results of this study demonstrate the influence of disclosed items on student learning and learning outcomes and suggest that a carefully implemented assessment is able to improve conceptual knowledge in physiology.

Use of very short answer questions compared to multiple choice questions in undergraduate medical students: An external validation study

Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

Is there ever a single best answer (SBA): assessment driving certainty in the uncertain world of GP?

Uncertainty is inherent in all areas of medical practice, not least in primary care, which is defined by its acceptance of uncertainty and complexity. Single best answer (SBA) questions are a ubiquitous assessment tool in undergraduate medical assessments; however clinical practice, particularly in primary care, challenges the supposition that a single best answer exists for all clinical encounters and dilemmas. In this article, we seek to highlight several aspects of the relationship between this assessment format and clinical uncertainty by considering its influence on medical students' views of uncertainty in the contexts of their medical education, personal epistemology, and clinical expectations.

Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

Very Short Answer Questions in Team-Based Learning: Limited Effect on Peer Elaboration and Memory

INTRODUCTION

In team-based learning (TBL), single best answer questions (SBAQs) are traditionally used because immediate computer-assisted feedback facilitates team discussions. Recent improvements in digital marking systems and criticisms of non-analytical strategies in SBAQs have prompted the consideration of very short answer questions (VSAQs) as an alternative to SBAQs. We aim to compare the effect of VSAQs and SBAQs on peer elaboration and knowledge retention in TBL.

MATERIALS AND METHODS

Twenty-four second-year students from the Lee Kong Chian School of Medicine participated in a mixed-methods study that included a randomised controlled crossover trial with two intervention arms (TBL-VSAQs and TBL-SBAQs). Two TBL sessions were conducted, with one topic covered each. Students were randomly allocated into six teams of four members. Individual and team scores and completion times were measured, and students were surveyed on their TBL experience. A follow-up quiz on the same topics was administered two weeks later to assess knowledge retention.

RESULTS

Individuals scored lower for VSAQs than SBAQs in the second TBL topic (7.17 ± 1.52 versus 8.25 ± 1.48; p = 0.046), while findings in other metrics were non-significant. Follow-up quiz scores showed no significant difference in knowledge retention, although effect size and power were low. Students perceived VSAQs as more authentic and challenging, though most preferred the continued use of SBAQs for TBL.

DISCUSSION

VSAQs have a limited impact on peer elaboration and knowledge retention versus SBAQs in TBL.

CONCLUSIONS

We should be circumspect about implementing VSAQs in TBL especially for medical students at the pre-clinical level.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-022-01716-5.

Oral presentation assessment and image reading behaviour on brain computed tomography reading in novice clinical learners: an eye-tracking study

BACKGROUND

To study whether oral presentation (OP) assessment could reflect the novice learners' interpretation skills and reading behaviour on brain computed tomography (CT) reading.

METHODS

Eighty fifth-year medical students were recruited, received a 2-hour interactive workshop on how to read brain CT, and were assigned to read two brain CT images before and after instruction. We evaluated their image reading behaviour in terms of overall OP post-test rating, the lesion identification, and competency in systematic image reading after instruction. Students' reading behaviour in searching for the target lesions were recorded by the eye-tracking technique and were used to validate the accuracy of lesion reports. Statistical analyses, including lag sequential analysis (LSA), linear mixed models, and transition entropy (TE) were conducted to reveal temporal relations and spatial complexity of systematic image reading from the eye movement perspective.

RESULTS

The overall OP ratings [pre-test vs. post-test: 0 vs. 1 in case 1, 0 vs. 1 in case 2, p < 0.001] improved after instruction. Both the scores of systematic OP ratings [0 vs.1 in both cases, p < 0.001] and eye-tracking studies (Case 1: 3.42 ± 0.62 and 3.67 ± 0.37 in TE, p = 0.001; Case 2: 3.42 ± 0.76 and 3.75 ± 0.37 in TE, p = 0.002) showed that the image reading behaviour changed before and after instruction. The results of linear mixed models suggested a significant interaction between instruction and area of interests for case 1 (p < 0.001) and case 2 (p = 0.004). Visual attention to the target lesions in the case 1 assessed by dwell time were 506.50 ± 509.06 and 374.38 ± 464.68 milliseconds before and after instruction (p = 0.02). However, the dwell times in the case 2, the fixation counts and the frequencies of accurate lesion diagnoses in both cases did not change after instruction.

CONCLUSION

Our results showed OP performance may change concurrently with the medical students' reading behaviour on brain CT after a structured instruction.

Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?

BACKGROUND

We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations.

METHODS

Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set.

RESULTS

A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z = -2.82, p = 0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z = 2.46, p = 0.014).

CONCLUSIONS

When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.

Survival Radiology: How a popular in-person interactive medical student radiology workshop pivoted online during the COVID-19 pandemic

Background

Survival Radiology (SR) is a flagship annual full-day in-person radiology workshop targeted at final year medical students in Singapore to prepare them for internship. Previous in-person editions have consistently received positive reviews from 2014 to 2019. However, the COVID-19 pandemic necessitated a rapid online pivot for its sixth edition in 2020.

Objectives

This study aims to (a) identify key success factors of a traditional in-person medical student radiology workshop, (b) describe the rapid online pivot in 2020 and (c) to identify key success factors for online educational initiatives.

Methods

Post-workshop survey responses of SR from 2014 to 2020 were evaluated. Likert-scale data were quantitatively analysed, while free-text responses were qualitatively analysed.

Results

A total of 1248 post-workshop surveys (2014–2020 workshops) and 266 free-text responses (2020 workshop) were received from 2640 participants over the years. Progressive changes that sustained or improved participant feedback for in-person SR workshops included adoption of a case-based approach, utility of ‘live’ audience response systems and incorporation of quizzes with a favourable overall feedback rating of 4.42–4.89 from 2014 to 2019. The webinar version of SR in 2020 became the best-rated edition since inception with a rating of 4.9. Qualitative analysis of feedback from SR 2020 showed that the participants preferred the webinar model, online modes of engagement and interactivity.

Conclusion

Our experience shows that it is not only possible to successfully pivot online for such workshops, but that blended educational formats utilising online engagements supplemented by in-person activities will be well-received by ‘Generation Z’ learners even after the COVID-19 pandemic.

Effectiveness of Large-Scale Preparedness Training on Electrocardiogram for Medical, Surgical, and Pre-clinical Doctors: A Need-Based Initiative for COVID-19 Patient Care

Introduction: One of the competencies expected of all doctors posted in coronavirus disease 2019 (COVID-19) wards, is ECG rhythm identification, interpretation, and intervention for immediate management of patients. This study was undertaken to evaluate the effectiveness of the ECG training module as a component of preparedness training to combat COVID-19.

Methodology: This was a cross-sectional study conducted during training on ECG rhythm identification, interpretation, its management in COVID-19 patients. Study participants included faculty, senior residents, junior residents, and interns of medical, surgical, and paraclinical disciplines. The training session included one hour of didactic lecture and one and half hours of interactive session during which case scenarios were discussed. An objective assessment was conducted through pre-test and post-test. Mean of pre and post-test scores were compared using paired t-test for evaluating statistical significance. Feedback was also taken from participants.

Results: Out of the 800 participants who gave consent, only 682 who completed both pre and post-test were included in the final analysis. Mean pre-test and post-test scores were 9.29/15 (61.9%) and 11.63/15 (77.5%), respectively, with a mean improvement of +2.34/15 (+15.6%). Of the participants, 38.6% obtained low scores in pre-test and 82% of respondents agreed that knowledge and skills gained from training would be useful in providing patient care.

Conclusion: Low baseline knowledge on ECG highlights the need for re-training doctors posted in COVID-19 care on cardiac rhythm identification and interpretation. Interactive training is effective in improving ECG interpretative skills among doctors across disciplines and is the appropriate method to retrain/reskill, especially for large-scale capacity building.

Transforming MRCPsych theory examinations: digitisation and very short answer questions (VSAQs)

Many examinations are now delivered online using digital formats, the migration to which has been accelerated by the COVID-19 pandemic. The MRCPsych theory examinations have been delivered in this way since Autumn 2020. The multiple choice question formats currently in use are highly reliable, but other formats enabled by the digital platform, such as very short answer questions (VSAQs), may promote deeper learning. Trainees often ask for a focus on core knowledge, and the absence of cueing with VSAQs could help achieve this. This paper describes the background and evidence base for VSAQs, and how they might be introduced. Any new question formats would be thoroughly piloted before appearing in the examinations and are likely to have a phased introduction alongside existing formats.

Using very short answer errors to guide teaching

Background

Student performance in examinations reflects on both teaching and student learning. Very short answer questions require students to provide a self‐generated response to a question of between one and five words, which removes the cueing effects of single best answer format examinations while still enabling efficient machine marking. The aim of this study was to pilot a method of analysing student errors in an applied knowledge test consisting of very short answer questions, which would enable identification of common areas that could potentially guide future teaching.

Methods

We analysed the incorrect answers given by 1417 students from 20 UK medical schools in a formative very short answer question assessment delivered online.

Findings

The analysis identified four predominant types of error: inability to identify the most important abnormal value, over or unnecessary investigation, lack of specificity of radiology requesting and over‐reliance on trigger words.

Conclusions

We provide evidence that an additional benefit to the very short answer question format examination is that analysis of errors is possible. Further assessment is required to determine if altering teaching based on the error analysis can lead to improvements in student performance.

Thinking differently - Students' cognitive processes when answering two different formats of written question

BACKGROUND

Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students' cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions.

METHODS

We undertook a 'think aloud' study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used.

RESULTS

Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students.

CONCLUSIONS

Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.

Evaluation of an online case-based learning module that integrates basic and clinical sciences

OBJECTIVE

Case-based online modules can be created to integrate basic science and clinical science knowledge. An integrated module was designed, implemented, and evaluated for student performance and perception.

METHODS

Five faculty members from both basic science and clinical education departments developed an integrative, online, case-based learning module. The case involved a patient with facial asymmetry, and the student was led to a diagnosis of Bell's palsy. Material on Bell's palsy was presented in an integrated module in addition to traditional lecture for a fall cohort of students and was presented only in traditional lecture format to a winter cohort of students. Both cohorts were given the same 5 multiple-choice questions on Bell's palsy as part of a midterm exam, and the scores of these test questions were compared between cohorts. A 13-question, mixed-methods survey was given to the fall cohort to determine their perceptions of the module and their learning.

RESULTS

Multiple-choice test question performance was equivalent between cohorts for the Bell's palsy questions (fall 2018: mean = 3.68, SD = 0.99; winter 2019: mean = 3.51, SD = 0.92). Perception survey responses indicated that students felt positively about the integrated module and that it was applicable and helpful with improving, reinforcing, and integrating basic science and clinical knowledge.

CONCLUSION

This study provides evidence that case-based integrated modules are perceived favorably by students and result in similar exam question performance. They can be a useful tool to help students connect information throughout the chiropractic curriculum.

Nutrition content of summative examinations within an Australian 4-year graduate entry medical course: 2013-2016

Background

Poor nutrition is a major contributor to chronic disease, but the level of nutrition education in medical training is limited. Deakin University Medical School has been working to embed more nutrition into the curriculum since 2009.

Aim

To assess the nutrition content of all summative examinations in the Bachelor of Medicine, Bachelor of Surgery over a 4-year period.

Methods

The type, amount and scope of nutrition-related questions were assessed in all summative examinations delivered to all 4-year levels from 2013 to 2016. These were assessed independently and analysed for nutrition content. The amount of nutrition was quantified, and the nutrition topic areas and nutrition competencies addressed were documented.

Results

Less than 10% of summative examination questions contained any nutrition content. For first-year and second-year students, these examinations included an average yearly total of 433 multiple choice questions (MCQs) (range 337–530) and 47 short answer questions (SAQs) (range 33–62). Third-year and fourth-year students had 150 MCQs on average per year and no SAQs. The percentage of nutrition-related questions across all 4 years ranged from 6% in 2013 to 10% in 2016. The proportion of SAQs with nutrition content ranged from 12% in 2013 to 19% in 2016. Basic nutritional sciences, accounted for 60% of nutrition content and, 25% addressed dietary strategies for prevention and treatment of disease, and skills-based nutrition competencies represented approximately 10% of all questions.

Summary and conclusions

Minimal nutrition was included in the summative examinations. There did not appear to be any consistent increase in the nutrition content of MCQs over the 4-year period but there was some indication of an increase in nutrition content in SAQs. Longer term evaluation is required to confirm this trend. Only a small number of nutrition questions were skills based, most focused on basic nutritional science. Examinations included few skills-based nutrition questions, and consideration of setting a minimum level of nutrition in examinations could assist in ensuring the development of appropriate nutrition competencies in medical graduates.

Exploring the feasibility of using very short answer questions (VSAQs) in team-based learning (TBL)

BACKGROUND

Team-based learning (TBL) currently relies on single best answer questions (SBAQs) to provide immediate feedback. Very short answer questions (VSAQs) are a reliable and discriminatory alternative that encourage learners to use more authentic clinical reasoning strategies compared to SBAQs. However, the challenge of marking VSAQs has limited their integration into TBL; we therefore explored the feasibility of VSAQs within a TBL session.

METHODS

An online platform was developed to allow immediate marking of VSAQs during the TBL sessions. As part of the readiness assurance process, students completed VSAQs and SBAQs, which were marked in real time.

RESULTS

Instructors were able to mark all VSAQs during the individual readiness assurance test (iRAT), which facilitated the provision of immediate feedback during the team readiness assurance test (tRAT). The mean time to mark five VSAQs was 422 seconds (SD 73 seconds). For VSAQs, the number of attempts to reach the correct answer ranged from 1 to 38, compared to 1 to 4 for SBAQs. In total, 71.6% of students agreed that using VSAQs in TBL helped to emphasise group discussions.

DISCUSSION

The wide range of attempts at, and students' perspectives of VSAQs are suggestive of their positive impact on student discussion during TBL. We demonstrate how new technology allows VSAQs to be feasibly integrated into TBL with the potential to enrich group discussions.

Question-Based Collaborative Learning for Constructive Curricular Alignment

INTRODUCTION

We designed a curriculum mapping tool which enables medical students to access intended learning outcomes (ILOs) on their iPads in the workplace. Students were encouraged to use the online curriculum map in a specially planned teaching session: question-based collaborative learning (QBCL). The aim of the session was to empower medical students to constructively align their experiential learning with the learning outcomes of the undergraduate curriculum. In doing so, our session aimed to provide students with a greater understanding of the curriculum, improve their insights into assessment and their question-writing abilities.

METHODS

The QBCL pre-session preparation involved reviewing a patient with a presentation that aligned to the year-specific ILOs. During a 150 minute QBCL session, students received training on how to write high quality multiple choice questions (MCQs) delivered by a faculty member of Imperial College School of Medicine. They then worked collaboratively in groups and created MCQs based on their clinical encounters. Their questions were tagged to the relevant learning objective and submitted online via the curriculum map. The student-generated MCQs were analyzed using an adjusted version of Bloom's taxonomy. We also conducted a quantitative evaluation of the session.

RESULTS

One hundred and sixty-three questions were submitted, with 81% of questions being tagged to ILOs considered to show evidence of learning consistent with the "Apply" tier of Bloom's taxonomy. The majority of students agreed that the session was interactive (80%), thought-provoking (77%) and improved their team-working skills (70%). It gave them a greater understanding of the undergraduate curriculum (65%), improved their question-writing and insight into assessments (76%), and provided an opportunity to learn from their peers (86%). Students agreed that this session covered a variety of cases (82%) and deepened their understanding of medical conditions and presentations (87%).

CONCLUSION

We encouraged students to actively interact with the curriculum map. Students were able to achieve their own constructive alignment by writing assessment items based on real patients and linking them to the appropriate intended learning outcomes.

Situational judgement test validity for selection: A systematic review and meta-analysis

CONTEXT

Situational judgement tests (SJTs) are widely used to evaluate 'non-academic' abilities in medical applicants. However, there is a lack of understanding of how their predictive validity may vary across contexts. We conducted a systematic review and meta-analysis to synthesise existing evidence relating to the validity of such tools for predicting outcomes relevant to interpersonal workplace performance.

METHODS

Searches were conducted in relevant databases to June 2019. Study quality and risk of bias were assessed using the Quality In Prognosis Studies (QUIPS) tool. Results were pooled using random effects meta-analysis and meta-regressions.

RESULTS

Initially, 470 articles were identified, 218 title or abstracts were reviewed, and 44 full text articles were assessed with 30 studies meeting the final inclusion criteria and were judged, overall, to be at moderate risk of bias. Of these, 26 reported correlation coefficients relating to validity, with a pooled estimate of 0.32 (95% confidence interval 0.26 to 0.39, P < .0001). Considerable heterogeneity was observed (I²  = 96.5%) with the largest validity coefficients tending to be observed for postgraduate, rather than undergraduate, selection studies (β = 0.23, 0.11 to 0.36, P < .001). The correction of validity coefficients for attenuation was also independently associated with larger effects (β = 0.13, 0.03 to 0.23, P = .01). No significant associations with test medium (video vs text format), cross-sectional study design, or period of assessment (one-off vs longer-term) were observed. Where reported, the scores generally demonstrated incremental predictive validity, over and above tests of knowledge and cognitive ability.

CONCLUSIONS

The use of SJTs in medical selection is supported by the evidence. The observed trend relating to training stage requires investigation. Further research should focus on developing robust criterion-relevant outcome measures that, ideally, capture interpersonal aspects of typical workplace performance. This will facilitate additional work identifying the optimal place of SJTs within particular selection contexts and further enhancing their effectiveness.

Very short answer questions: a viable alternative to multiple choice questions

BACKGROUND

Multiple choice questions, used in medical school assessments for decades, have many drawbacks such as hard to construct, allow guessing, encourage test-wiseness, promote rote learning, provide no opportunity for examinees to express ideas, and do not provide information about strengths and weakness of candidates. Directly asked, directly answered questions like Very Short Answer Questions (VSAQ) are considered a better alternative with several advantages.

OBJECTIVES

This study aims to compare student performance in MCQ and VSAQ and obtain feedback. from the stakeholders.

METHODS

Conduct multiple true-false, one best answer, and VSAQ tests in two batches of medical students, compare their scores and psychometric indices of the tests and seek opinion from students and academics regarding these assessment methods.

RESULTS

Multiple true-false and best answer test scores showed skewed results and low psychometric performance compared to better psychometrics and more balanced student performance in VSAQ tests. The stakeholders' opinions were significantly in favour of VSAQ.

CONCLUSION AND RECOMMENDATION

This study concludes that VSAQ is a viable alternative to multiple-choice question tests, and it is widely accepted by medical students and academics in the medical faculty.