Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions

PROBLEM

Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students' performance on LLM-generated questions to questions developed by humans.

APPROACH

The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT.

OUTCOMES

The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly.

NEXT STEPS

Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.

Position statement of the GMA committee "teaching evaluation"

The evaluation of teaching can be an essential driver for curriculum development. Instruments for teaching evaluation are not only used for the purpose of quality assurance but also in the context of medical education research. Therefore, they must meet the common requirements for reliability and validity. This position paper from the GMA Teaching Evaluation Committee discusses strategic and methodological aspects of evaluation in the context of undergraduate medical education and related courses; and formulates recommendations for the further development of evaluation. First, a four-step approach to the design and implementation of evaluations is presented, then methodological and practical aspects are discussed in more detail. The focus here is on target and confounding variables, survey instruments as well as aspects of implementation and data protection. Finally, possible consequences from evaluation data for the four dimensions of teaching quality (structural and procedural aspects, teachers and outcomes) are discussed.

Medical students' AI literacy and attitudes towards AI: a cross-sectional two-center study using pre-validated assessment instruments

BACKGROUND

Artificial intelligence (AI) is becoming increasingly important in healthcare. It is therefore crucial that today's medical students have certain basic AI skills that enable them to use AI applications successfully. These basic skills are often referred to as "AI literacy". Previous research projects that aimed to investigate medical students' AI literacy and attitudes towards AI have not used reliable and validated assessment instruments.

METHODS

We used two validated self-assessment scales to measure AI literacy (31 Likert-type items) and attitudes towards AI (5 Likert-type items) at two German medical schools. The scales were distributed to the medical students through an online questionnaire. The final sample consisted of a total of 377 medical students. We conducted a confirmatory factor analysis and calculated the internal consistency of the scales to check whether the scales were sufficiently reliable to be used in our sample. In addition, we calculated t-tests to determine group differences and Pearson's and Kendall's correlation coefficients to examine associations between individual variables.

RESULTS

The model fit and internal consistency of the scales were satisfactory. Within the concept of AI literacy, we found that medical students at both medical schools rated their technical understanding of AI significantly lower (MMS1 = 2.85 and MMS2 = 2.50) than their ability to critically appraise (MMS1 = 4.99 and MMS2 = 4.83) or practically use AI (MMS1 = 4.52 and MMS2 = 4.32), which reveals a discrepancy of skills. In addition, female medical students rated their overall AI literacy significantly lower than male medical students, t(217.96) = -3.65, p <.001. Students in both samples seemed to be more accepting of AI than fearful of the technology, t(745.42) = 11.72, p <.001. Furthermore, we discovered a strong positive correlation between AI literacy and positive attitudes towards AI and a weak negative correlation between AI literacy and negative attitudes. Finally, we found that prior AI education and interest in AI is positively correlated with medical students' AI literacy.

CONCLUSIONS

Courses to increase the AI literacy of medical students should focus more on technical aspects. There also appears to be a correlation between AI literacy and attitudes towards AI, which should be considered when planning AI courses.

Game design elements of serious games in the education of medical and healthcare professions: a mixed-methods systematic review of underlying theories and teaching effectiveness

Serious games, as a learning resource, enhance their game character by embedding game design elements that are typically used in entertainment games. Serious games in its entirety have already proven their teaching effectiveness in different educational contexts including medical education. The embedded game design elements play an essential role for a game's effectiveness and thus they should be selected based on evidence-based theories. For game design elements embedded in serious games used for the education of medical and healthcare professions, an overview of theories for the selection lacks. Additionally, it is still unclear whether and how single game design elements affect the learning effectiveness. Therefore, the main aim of this systematic review is threefold. Firstly, light will be shed on the single game design elements used in serious games in this area. Second, the game design elements' underlying theories will be worked out, and third, the game design elements' effectiveness on student learning outcome will be assessed. Two literature searches were conducted in November 2021 and May 2022 in six literature databases with keywords covering the fields of educational game design, serious game, and medical education. Out of 1006 initial records, 91 were included after applying predefined exclusion criteria. Data analysis revealed that the three most common game design elements were points, storyline, and feedback. Only four underlying theories were mentioned, and no study evaluated specific game design elements. Since game design elements should be based on theories to ensure meaningful evaluations, the conceptual GATE framework is introduced, which facilitates the selection of evidence-based game design elements for serious games.

[Smoking cessation in hospitalised patients - Initiate among inpatients, continue when outpatients - A Position Paper by the German Respiratory Society (DGP) Taskforce for Smoking Cessation]

Tobacco smoking is the greatest preventable health risk. The effects are serious, both individually and societal. Nevertheless, the current prevalence of tobacco smokers in Germany is still high at around 35 %. A recent strong increase in actively smoking adolescents (14- to 17-year-olds, current prevalence approx. 16 %) and young adults (18- to 24-year-olds, current prevalence approx. 41 %) is also a cause for concern. About a third of all inpatients continue smoking while being treated. The hospitalization of active smokers in acute and rehabilitation hospitals serves as a "teachable moment" for initiation of cessation offers. An intervention that begins in hospital and continues for at least a month after discharge results in about 40 % additional smokefree patients. It is scientifically well-researched, effective and cost-efficient. After initiation in hospital these measures can be continued via ambulatory cessation programs, rehabilitation facilities, an Internet or telephone service. In Germany, there are structured and quality-assured cessation offers, both for the inpatient and for the outpatient area. The biggest obstacle to broad establishment of such offers is the lack of reimbursement. Two feasible ways to change this would be the remuneration of the existing OPS 9-501 "Multimodal inpatient treatment for smoking cessation" and the establishment of quality contracts according to § 110a SGB V. An expansion of tobacco cessation measures in healthcare facilities would reduce smoking prevalence, associated burden of disease and consecutive costs.

Conception and Feasibility of a Digital Tele-Guided Abdomen, Thorax, and Thyroid Gland Ultrasound Course for Medical Students (TELUS study)

PURPOSE

 Medical education has been transformed during the COVID-19 pandemic, creating challenges regarding adequate training in ultrasound (US). Due to the discontinuation of traditional classroom teaching, the need to expand digital learning opportunities is undeniable. The aim of our study is to develop a tele-guided US course for undergraduate medical students and test the feasibility and efficacy of this digital US teaching method.

MATERIALS AND METHODS

 A tele-guided US course was established for medical students. Students underwent seven US organ modules. Each module took place in a flipped classroom concept via the Amboss platform, providing supplementary e-learning material that was optional and included information on each of the US modules. An objective structured assessment of US skills (OSAUS) was implemented as the final exam. US images of the course and exam were rated by the Brightness Mode Quality Ultrasound Imaging Examination Technique (B-QUIET). Achieved points in image rating were compared to the OSAUS exam.

RESULTS

 A total of 15 medical students were enrolled. Students achieved an average score of 154.5 (SD ± 11.72) out of 175 points (88.29 %) in OSAUS, which corresponded to the image rating using B-QUIET. Interrater analysis of US images showed a favorable agreement with an ICC (2.1) of 0.895 (95 % confidence interval 0.858 < ICC < 0.924).

CONCLUSION

 US training via teleguidance should be considered in medical education. Our pilot study demonstrates the feasibility of a concept that can be used in the future to improve US training of medical students even during a pandemic.

Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods

BACKGROUND

Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data.

OBJECTIVE

This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6).

METHODS

Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05).

RESULTS

We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977).

CONCLUSIONS

Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees' scores and overall examination results.

Adventure Legal Medicine: a free online serious game for supplementary use in undergraduate medical education

Serious games (computer-based learning games) are increasingly used in medical education at various levels, as user access is independent of location and time and promotes non-linear learning. In legal medicine, interactive digital media are still scarce. The freely accessible online serious game "Adventure Legal Medicine" was developed as part of the "Hamburg Open Online University". The goal was to teach the basics of forensic casework in a point-and-click adventure setting consisting of five cases. During development, 40 medical students were asked to evaluate the game anonymously. The System Usability Scale (SUS) resulted in a mean score of 86.7 (SD 8.3), which corresponds to above-average usability. Further specific evaluations revealed a good to very good rating of the game with no differences in terms of gender (p = 0.214), first-year versus advanced students (p = 0.393) and students who never/rarely or sometimes/often played computer games (p = 0.780). Since there are only a few digital media so far that allow curricular integration into undergraduate teaching in legal medicine, this serious game represents a possibility to integrate digital media into both face-to-face teaching and distance learning and to use it as a supplement to the medical school's own teaching offer, encouraging users to actively engage with the subject.

Effects of Elaboration and Instructor Feedback on Retention of Clinical Reasoning Competence Among Undergraduate Medical Students: A Randomized Crossover Trial

This randomized crossover trial examines whether elaboration on common errors in patient treatment, combined with individualized mailed feedback, improves medium-term retention of clinical reasoning competence.

Effect of a flipped classroom course to foster medical students' AI literacy with a focus on medical imaging: a single group pre-and post-test study

BACKGROUND

The use of artificial intelligence applications in medicine is becoming increasingly common. At the same time, however, there are few initiatives to teach this important and timely topic to medical students. One reason for this is the predetermined medical curriculum, which leaves very little room for new topics that were not included before. We present a flipped classroom course designed to give undergraduate medical students an elaborated first impression of AI and to increase their "AI readiness".

METHODS

The course was tested and evaluated at Bonn Medical School in Germany with medical students in semester three or higher and consisted of a mixture of online self-study units and online classroom lessons. While the online content provided the theoretical underpinnings and demonstrated different perspectives on AI in medical imaging, the classroom sessions offered deeper insight into how "human" diagnostic decision-making differs from AI diagnoses. This was achieved through interactive exercises in which students first diagnosed medical image data themselves and then compared their results with the AI diagnoses. We adapted the "Medical Artificial Intelligence Scale for Medical Students" to evaluate differences in "AI readiness" before and after taking part in the course. These differences were measured by calculating the so called "comparative self-assessment gain" (CSA gain) which enables a valid and reliable representation of changes in behaviour, attitudes, or knowledge.

RESULTS

We found a statistically significant increase in perceived AI readiness. While values of CSA gain were different across items and factors, the overall CSA gain regarding AI readiness was satisfactory.

CONCLUSION

Attending a course developed to increase knowledge about AI in medical imaging can increase self-perceived AI readiness in medical students.

Publication activities relating to digital teaching and learning in the GMS Journal for Medical Education - a descriptive analysis (1984-2020)

AIMS AND OBJECTIVES

Digital teaching, learning and assessment have been part of medical education and continuing education for decades. The objective of this review paper is to highlight developments and perspectives in these areas in the GMS Journal for Medical Education (GMS JME).

METHODOLOGY

In the spring of 2020, we conducted a systematic literature search of the Journal for Medical Education (JME) and analysed the articles with regard to different categories such as article type, digital tools used or mode of data collection.

RESULTS

Of the 132 articles analysed, 78 were digital interventions (53 of which were exploratory-descriptive), 28 were project descriptions, 16 were surveys of needs or equipment and 10 were concept papers. About one-third of the studies and project reports each dealt with virtual patients or case-based learning, whereas no articles were published on trends such as serious games or virtual reality. Overall, our analysis shows that in many respects, the studies on digital teaching were more broadly based, especially between 2006 and 2010, after which this trend tended to decline again.

CONCLUSIONS

Our analysis shows that publications in the JME consider some key aspects of digital teaching in medical education and continuing education, such as educational videos or virtual patients. The variability of information and methods of presentation advocate the use of guidelines to optimise the quality of scientific papers. Furthermore, clues for future research topics and experimental study designs are identified.

Using item response theory to appraise key feature examinations for clinical reasoning

BACKGROUND

Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach.

METHODS

Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach's alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function.

RESULTS

According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges.

CONCLUSIONS

Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.

Considering vocational training as selection criterion for medical students: evidence for predictive validity

Prior work experience in a relevant medical profession is an important admission criterion currently used at many German medical schools in addition to cognitive criteria. In other countries, work experience is often considered in later admission stages (e.g., interviews with pre-selected subgroups of applicants). However, evidence for its predictive validity for study success in addition to cognitive admission criteria is currently lacking. We therefore assessed whether completed vocational training in a relevant medical profession can predict study performance in the first two years of study in addition to cognitive admission criteria. Admission and study performance data of all currently enrolled medical students at two German medical schools (Göttingen and Heidelberg) beginning with the 2013/14 cohort were retrospectively analyzed. Cognitive admission criteria in our sample were GPA grades and a cognitive test ("Test für Medizinische Studiengänge", TMS). We defined the study outcome parameter as the mean percentile rank over all performance data points over the first two years of study for each location, respectively. A multi-level model with varying intercepts by location, GPA, TMS, vocational training, and sex as predictors accounted for 14.5% of the variance in study outcome. A positive predictive association with study outcome was found for vocational training (ß = 0.33, p = .008) beyond GPA (ß = 0.38, p < .001) and TMS (ß = 0.26, p < .001). Our results support the use of prior vocational training as a selection criterion for medical studies potentially adding predictive validity to cognitive criteria.

Digital patient-centred learning in medical education: A national learning platform with virtual patients as part of the DigiPaL project

Background: Due to the coronavirus pandemic, the medical faculties in the Federal Republic of Germany converted their curricula to digital formats on a large scale and very quickly in spring 2020 as an emergency measure. At the same time, a start was made on the nationwide exchange of digital teaching/learning materials via the online platform "LOOOP share" in order to save local resources. Among other things, virtual patient cases (VP) were shared across faculties for case-based learning, through which students can acquire clinical decision-making skills. Objectives: Within the framework of the cooperation project "National Learning Platforms for Digital Patient-Related Learning in Medical Studies" (DigiPaL), the usability of VPs for students and teachers should be improved, and the spectrum of disease patterns that are covered by VPs should be systematically expanded. Results: With the participation of many locations, a total of 150 VPs were developed by 96 case authors from 16 faculties, in addition to the existing 403 VPs. The thematic selection was made on the basis of criteria oriented to the National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM). After completion, these VPs were also made available to all faculties for free use via "LOOOP share" and the CASUS learning platform. Discussion: Even after the pandemic, these developed VPs should be available to the faculties and thus make a lasting contribution to improve medical training in Germany - especially in light of digital teaching formats being expressly advocated on the basis of the adapted current Medical Licensure Act (ÄApprO). A possible application is interdisciplinary learning of clinical decision-making with the help of blended learning formats within the framework of a longitudinal curriculum. The large number of involved colleagues and faculties shows that the nationally coordinated development of VPs across faculties was commonly seen as useful.

Use of a Serious Game to Teach Infectious Disease Management in Medical School: Effectiveness and Transfer to a Clinical Examination

Purpose

Physicians of all specialties must be familiar with important principles of infectious diseases, but curricular time for this content is limited and clinical teaching requires considerable resources in terms of available patients and teachers. Serious games are scalable interventions that can help standardize teaching. This study assessed whether knowledge and skills acquired in a serious game translate to better performance in a clinical examination.

Methods

Fifth-year undergraduate medical students (n = 100) at Goettingen Medical School were randomized to three groups receiving different levels of exposure to virtual patients presenting with signs and symptoms of either infective endocarditis or community-acquired pneumonia in a serious game simulating an accident and emergency department. Student performance was assessed based on game logfiles and an objective standardized clinical examination (OSCE).

Results

Higher exposure to virtual patients in the serious game did not result in superior OSCE scores. However, there was good agreement between student performance in the OSCE and in game logfiles (r = 0.477, p = 0.005). An Item Response Theory analysis suggested that items from the serious game covered a wider range of ability, thus better differentiating between students within a given cohort.

Conclusion

Repeated exposure to virtual patients with infectious diseases in a serious game did not directly impact on exam performance but game logfiles might be good and resource-sparing indicators of student ability. One advantage of using serious games in medical education is standardized content, a lower inhibition threshold to learn, and a need of less staff time compared to small-group clinical teaching.

Use of digital teaching resources and predictors of medical student performance during the pandemic: A prospective study

Background

The coronavirus pandemic has led to increased use of digital teaching formats in medical education. A number of studies have assessed student satisfaction with these resources. However, there is a lack of studies investigating changes in student performance following the switch from contact to virtual teaching. Specifically, there are no studies linking student use of digital resources to learning outcome and examining predictors of failure.

Methods

Student performance before (winter term 2019/20: contact teaching) and during (summer term 2020: no contact teaching) the pandemic was compared prospectively in a cohort of 162 medical students enrolled in the clinical phase of a five-year undergraduate curriculum. Use of and performance in various digital resources (case-based teaching in a modified flipped classroom approach; formative key feature examinations of clinical reasoning; daily multiple choice quizzes) was recorded in summer 2020. Student scores in summative examinations were compared to examination scores in the previous term. Associations between student characteristics, resource use and summative examination results were used to identify predictors of performance.

Results

Not all students made complete use of the digital learning resources provided. Timely completion of tasks was associated with superior performance compared to delayed completion. Female students scored significantly fewer points in formative key feature examinations and digital quizzes. Overall, higher rankings within the student cohort (according to summative exams) in winter term 2019/20 as well as male gender predicted summative exam performance in summer 2020. Scores achieved in the first formative key feature examination predicted summative end-of-module exam scores.

Conclusions

The association between timely completion of tasks as well as early performance in a module and summative exams might help to identify students at risk and offering help early on. The unexpected gender difference requires further study to determine whether the shift to a digital-only curriculum disadvantages female students.

Potential Explanations for Conflicting Findings on Abrupt Versus Gradual Smoking Cessation: A Population Study in England

INTRODUCTION

Observational and trial evidence conflict on the efficacy of two contrasting behavioral approaches to quitting smoking-gradual and abrupt. Observational data suggest an abrupt approach to quitting is superior to a gradual approach, whilst trials show no difference. One potential explanation is self-selection in observational data, whereby people can choose their quit approach, and those who find it harder to quit may be more likely to choose a gradual quit approach. This study aims to investigate potential explanations for these conflicting findings.

AIMS AND METHODS

This study aims to investigate potential explanations for these conflicting findings. We used observational data from a nationally representative sample of adults in England from November 2006 to February 2020 who reported smoking and had made at least one quit attempt in the past year (n = 21 542). We used logistic regression models to assess the association between abrupt versus gradual quit attempts and quit success, adjusting for sociodemographic, smoking, and quit attempt characteristics.

FINDINGS

Abrupt, versus gradual, attempts were associated with improved quit success in an unadjusted model (odds ratio = 2.02, 95% CI = 1.86 to 2.19). This association remained after adjusting for a broad range of relevant confounders (odds ratio = 1.75, 95% CI = 1.59 to 1.93).

CONCLUSIONS

Among a representative sample of adults who had smoked and made a quit attempt in the past year, there was evidence of an association between abrupt attempts and quit success before and after adjusting for relevant confounders. This suggests that the differences in quit success seen between abrupt and gradual quit attempt types are not completely driven by self-selection in observational data.

IMPLICATIONS

We investigated explanations for conflicting findings on the efficacy of gradual versus abrupt approaches to quitting smoking between trial and observational data. Despite adjusting observational data for sociodemographic, smoking, and quit attempt characteristics, an association between abrupt quitting and quit success remained. Therefore, differences in quit success were not completely driven by the self-selection of a gradual approach by people who found it especially difficult to quit or differences in the use of quitting aids. However, characteristics adjusted for were limited by the data available, and future research should continue to investigate the difference in findings across study types to inform cessation support.

Students' Perspectives on Curricular Ultrasound Education at German Medical Schools

Background: Despite ultrasound being an inherent part of medical education, only a few German medical schools have established a comprehensive ultrasound curriculum. This study aimed to explore medical students' perspectives on ultrasound in medical education (USMed).

Results: Between January 1^st, 2019 und June 30^th, 2019, an online survey was conducted among German medical students via the students' associations and their respective teaching facilities. The survey consisted of 17 items regarding USMed. Statements were rated on a 4-point Likert scale for agreement. In total, 1040 students from 31 German medical faculties participated. The majority (1021, 98.2%) reported a very high to high interest in curricular USMed. Students agreed (n = 945, 90.9%) that USMed would be helpful along their entire course of medical studies. Considering the best starting time for USMed, the opinions of German medical students diverged: students studying in a model curriculum preferred to start in the second year (40.7%) while 49% of the students studying in a traditional curriculum preferred to start in the third year (p ≤ 0.001). An insufficient allotment of time for USMed in the planned curriculum (675, 65%) and a lack of courses run by medical faculty (305, 29.4%) were listed as perceived significant barriers to the participation in USMed. Peer teaching was regarded as an effective method in realizing USMed by 731 (70.3%) students.

Conclusion: German medical students are very interested and willing to participate in USMed. There appears to be a high demand for US courses offered by medical schools.

A Weighted Evaluation Study of Clinical Teacher Performance at Five Hospitals in the UK

INTRODUCTION

Evaluation of individual teachers in undergraduate medical education helps clinical teaching fellows identify their own strengths and weaknesses. In addition, evaluation data can be used to guide career decisions. In order for evaluation results to adequately reflect true teaching performance, a range of parameters should be considered when designing data collection tools.

METHODS

Clinical teaching fellows at five London teaching hospitals were evaluated by third-year students they had supervised during a ten-week clinical attachment. The questionnaire addressed (a) general teaching skills and (b) student learning outcome measured via comparative self-assessments. Teachers were ranked using different algorithms with various weights assigned to these two factors.

RESULTS

A total of 133 students evaluated 14 teaching fellows. Overall, ratings on teaching skills were largely favourable, while the perceived increase in student performance was modest. Considerable variability across teachers was observed for both factors. Teacher rankings were strongly influenced by the weighting algorithm used. Depending on the algorithm, one teacher was assigned any rank between #2 and #10.

CONCLUSION

Both parts of the questionnaire address different outcomes and thus highlight specific strengths and weaknesses of individual teachers. Programme directors need to carefully consider the weight assigned to individual components of teacher evaluations in order to ensure a fair appraisal of teacher performance.

Training general practitioners in the ABC versus 5As method of delivering stop-smoking advice: a pragmatic, two-arm cluster randomised controlled trial

This study assessed the effectiveness of a 3.5-h training session for general practitioners (GPs) in providing brief stop-smoking advice and compared two methods of giving advice - ABC versus 5As - on the rates of delivery of such advice and of recommendations of evidence-based smoking cessation treatment during routine consultations. A pragmatic, two-arm cluster randomised controlled trial was carried out including a pre-/post-design for the analyses of the primary outcome in 52 GP practices in Germany. Practices were randomised (1:1) to receive a 3.5-h training session (ABC or 5As). In total, 1937 tobacco-smoking patients, who consulted trained GPs in these practices in the 6 weeks prior to or following the training, were included. The primary outcome was patient-reported rates of GP-delivered stop-smoking advice prior to and following the training, irrespective of the training method. Secondary outcomes were patient-reported receipt of recommendation/prescription of behavioural therapy, pharmacotherapy or combination therapy for smoking cessation, and the effectiveness of ABC versus 5As regarding all outcomes. GP-delivered stop-smoking advice increased from 13.1% (n=136 out of 1039) to 33.1% (n=297 out of 898) following the training (adjusted odds ratio (aOR) 3.25, 95% CI 2.34-4.51). Recommendation/prescription rates of evidence-based treatments were low (<2%) pre-training, but had all increased after training (e.g. behavioural support: aOR 7.15, 95% CI 4.02-12.74). Delivery of stop-smoking advice increased non-significantly (p=0.08) stronger in the ABC versus 5As group (aOR 1.71, 95% CI 0.94-3.12). A single training session in stop-smoking advice was associated with a three-fold increase in rates of advice giving and a seven-fold increase in offer of support. The ABC method may lead to higher rates of GP-delivered advice during routine consultations.

Quit attempts and tobacco abstinence in primary care patients: follow-up of a pragmatic, two-arm cluster randomised controlled trial on brief stop-smoking advice - ABC versus 5As

We developed a 3.5-h training for general practitioners (GPs) in delivering brief stop-smoking advice according to different methods (ABC, 5As). In a pragmatic, cluster randomised controlled trial our training proved effective in increasing GP-delivered rates of such advice (from 13% to 33%). In this follow-up analysis we examined the effect of the training and compared ABC versus 5As on patient-reported quit attempts and point prevalence abstinence at weeks 4, 12 and 26 following GP consultation. Follow-up data were collected in 1937 smoking patients - independently of the receipt of GP advice - recruited before or after the training of 69 GPs. At week 26, ∼70% of the patients were lost to follow-up. All 1937 patients were included in an intention-to-treat analysis; missing outcome data were imputed. Quit attempts and abstinence rates did not differ significantly from pre- to post-training or between patients from the ABC versus the 5As group. However, ancillary analyses showed that patients who received GP advice compared to those who did not had two times higher odds of reporting a quit attempt at all follow-ups and abstinence at week 26. We reported that our training increases GP-delivered rates of stop-smoking advice, and the present analysis confirms that advice is associated with increased quit attempts and abstinence rates in patients. However, our training did not further improve these rates, which might be related to patients' loss to follow-up or to contextual factors, e.g. access to free evidence-based cessation treatment, which can hamper the transfer of GPs' advice into patients' behaviour change.

AB0846 CONCEPTION AND FEASIBILITY OF A DIGITAL TELE-GUIDED ABDOMEN, THORAX AND THYROID GLAND ULTRASOUND COURSE FOR MEDICAL STUDENTS

Background:

Over the past few decades, technological advances in both ultrasound (US) and the application of telemedicine have been made [1]. Medical education has shifted to online classes during the COVID-19 pandemic, creating challenges in adequate training of US.

Objectives:

In the context of the current COVID-19 pandemic and the discontinuation of traditional classroom teaching, a tele-didactic US course for undergraduate medical students was developed. This study is examining the educational outcome of a new digital US course of the abdomen, thorax and the thyroid gland with the implementation of a modern portable US system.

Methods:

A tele-guided US course was established between April 1st and June 20th, 2020, at the University Hospital in Bonn, Germany. Students completed pre-and post-course surveys and underwent six US organ modules. Each module took place in a flipped-classroom concept including a digital learning platform. An objective structured assessment of US skills (OSAUS) [2] was implemented as final exam. Further, US images of the course and exam were rated by the Brightness Mode Quality Ultrasound Imaging Examination Technique (B-QUIET) [3]. Achieved points in image rating were compared to the OSAUS exam and survey results.

Results:

A total of 15 medical students were enrolled. There was a significant increase in self-assessed theoretical and practical ability (p < 0.001). Students achieved an average score of 154.5 (SD ± 11.72) out of 175 points (88.29%) in OSAUS, which corresponded to the image rating using B-QUIET and self-assessment data obtained in surveys. Interrater analysis of US images showed a favorable agreement with an ICC (2,1) of 0.895 (95% confidence interval 0.858 < ICC < 0.924).

Conclusion:

US training via teleguidance should be considered in medical education. Our pilot study demonstrates the feasibility of this concept that can be used in the future to improve US training of medical students even during a pandemic. The digital implementation with an affordable, portable point-of-care-US device could be an incredible opportunity to expedite US training worldwide.

References:

[1]Law J, Macbeth PB. Ultrasound: from Earth to space. Mcgill J Med 2011; 13(2): 59 [PMID: 22399873]

[2]Tolsgaard MG, Todsen T, Sorensen JL, et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. PLoS One 2013; 8(2): e57687

[https://doi.org/10.1371/journal.pone.0057687][PMID: 23469051]

[3]Bahner DP, Adkins EJ, Nagel R, Way D, Werman HA, Royall NA. Brightness mode quality ultrasound imaging examination technique (B-QUIET): quantifying quality in ultrasound imaging. J Ultrasound Med 2011; 30(12): 1649–55

[4][https://doi.org/10.7863/jum.2011.30.12.1649][PMID: 22124000]

Figure 1.

Implementation of the digital ultrasound course concept

Classroom setting is displayed, demonstrating simultaneous screen transmission of ultrasound (US) and camera images. A: The peer tutor’s camera image is shared with the students while demonstrating the US examination, B: At the same time the tutor’s mobile phone screen is shared with the students in order to visualize the US image acquisition, C: Students performing the US examination on their own, D: Example of a worksheet, which had to be completed within the app and was stored along with the US images in the cloud

Disclosure of Interests:

None declared

A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

BACKGROUND

Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.

METHODS

Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.

RESULTS

Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values.

CONCLUSIONS

Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

Effectiveness of a serious game addressing guideline adherence: cohort study with 1.5-year follow-up

BACKGROUND

Patients presenting with acute shortness of breath and chest pain should be managed according to guideline recommendations. Serious games can be used to train clinical reasoning. However, only few studies have used outcomes beyond student satisfaction, and most of the published evidence is based on short-term follow-up. This study investigated the effectiveness of a digital simulation of an emergency ward regarding appropriate clinical decision-making.

METHODS

In this prospective trial that ran from summer 2017 to winter 2018/19 at Göttingen Medical University Centre, a total of 178 students enrolled in either the fourth or the fifth year of undergraduate medical education took six 90-min sessions of playing a serious game ('training phase') in which they managed virtual patients presenting with various conditions. Learning outcome was assessed by analysing log-files of in-game activity (including choice of diagnostic methods, differential diagnosis and treatment initiation) with regard to history taking and patient management in three virtual patient cases: Non-ST segment elevation myocardial infarction (NSTEMI), pulmonary embolism (PE) and hypertensive crisis. Fourth-year students were followed up for 1.5 years, and their final performance was compared to the performance of students who had never been exposed to the game but had otherwise taken the same five-year undergraduate course.

RESULTS

During the training phase, overall performance scores increased from 57.6 ± 1.1% to 65.5 ± 1.2% (p < 0.001; effect size 0.656). Performance remained stable over 1.5 years, and the final assessment revealed a strong impact of ever-exposure to the game on management scores (72.6 ± 1.2% vs. 63.5 ± 2.1%, p < 0.001; effect size 0.811). Pre-exposed students were more than twice as likely to correctly diagnose NSTEMI and PE and showed significantly greater adherence to guideline recommendations (e.g., troponin measurement and D-dimer testing in suspected PE).

CONCLUSIONS

The considerable difference observed between previously exposed and unexposed students suggests a long-term effect of using the game although retention of specific virtual patient cases rather than general principles might partially account for this effect. Thus, the game may foster the implementation of guideline recommendations.

Restructuring the clinical curriculum at University Medical Center Göttingen: effects of distance teaching on students' satisfaction and learning outcome

Introduction: In summer term 2020, the clinical phase of the undergraduate medical curriculum at University Medical Center Göttingen was restructured since distance teaching had to be used predominantly due to contact restrictions during the COVID-19 pandemic. This paper investigates the impact of restructuring the clinical curriculum on medical students' satisfaction and learning outcomes. Methods: In each cohort, the 13-week curriculum was divided into two parts: During the first 9 weeks, factual knowledge was imparted using distance teaching by means of a modified inverted classroom approach. This was followed by a 4-week period of adapted classroom teaching involving both real and virtual patients in order to train students' practical skills. The evaluation of the 21 clinical modules comprised students' satisfaction with distance teaching as well as students' learning outcome. The latter was assessed by means of comparative self-assessment (CSA) gain and the results of the module exams, respectively. Data of summer term 2020 (= distance teaching, DT) were compared with respective data of winter term 2019/20 (= classroom teaching, CT) and analysed for differences and correlations. Results: Response rates of evaluations were 51.3% in CT and 19.3% in DT. There was no significant difference between mean scores in module exams in CT and DT, respectively. However, CSA gain was significantly lower in DT (p=0.047) compared with CT. Further analyses revealed that CSA gain depended on the time point of data collection: CSA gain was lower the more time had passed since the end of a specific module. Moreover, we found positive correlations between CSA gain and students' satisfaction with various aspects of distance teaching, particularly with "communication between teachers and students" (rho=0.674; p=0.002). Discussion and conclusions: Although some limitations and confounding factors have to be taken into account (such as evaluation response rates, assessment time points, and proportion of familiar items in module exams), the following recommendations can be derived from our findings: A valid assessment of students' learning outcome by means of exam results requires that as few exam items as possible are familiar to the students. CSA gain seems to be valid if assessment time points are standardised and not contaminated by students' learning activities for other modules. Good communication between teachers and students may contribute to increase students' satisfaction with distance teaching.

Tobacco Use, Knowledge about Smoking-Associated Risks, and Cessation Programs among Dental Students in Germany - ToDent

BACKGROUND

Professional and consistent smoking prevention and cessation may avoid many smoking-associated deaths worldwide. Dentists can exert a decisive influence on smoking behavior, as most people regularly visit a dentist. However, only if dentists have been trained sufficiently about the consequences of smoking and of smoking cessation methods might they fulfill this task appropriately.

OBJECTIVES

The aim of our study was therefore to examine the prevalence of smokers among dental students in Germany, their attitude toward smoking in general, and their knowledge about tobacco-related diseases and smoking cessation programs.

METHODS

A cross-sectional survey among dental students in Germany was conducted in 2016. Students of the first and tenth semesters were asked to participate in this anonymous survey. The questionnaire included items concerning their own smoking habits, their knowledge about effects of smoking on health, and their attitude toward prevention and cessation in a dental setting.

RESULTS

Seven hundred and thirty dental students participated in this survey; 21% of the participating students were smokers. Their knowledge about smoking-related diseases was widespread. Some diseases were rarely known (only in 34% of students) and others were well known (in 99% of the students). Knowledge about health risks of e-cigarettes was low. About 30% of the participating students felt competent to encourage and support patients in cessation strategies; 2 out of 3 students were willing to improve their knowledge in that area. More than 85% confirmed that advice for smoking cessation should be one of the tasks of a dentist.

CONCLUSIONS

Smoking is common among dental students, their knowledge about smoking-related health risks is moderate, and self-perceived smoking cessation skills are poor. More emphasis should be placed on education regarding smoking prevention and cessation methods.

Understanding the Association Between Spontaneous Quit Attempts and Improved Smoking Cessation Success Rates: A Population Survey in England With 6-Month Follow-up

INTRODUCTION

Almost half of smoking quit attempts are "spontaneous" (initiated as soon as the decision to quit has been made) and are associated with increased success rates. This study aimed to assess to what extent other factors may account for this association.

METHODS

Data were used from respondents to a survey representative of the adult population in England from 2006 to 2016. We included 2018 respondents who were current smokers at baseline and had attempted to quit between baseline and 6-month follow-up. Logistic regression models assessed the association between quit success and spontaneous quit attempts while adjusting for smoking, sociodemographic, and quit attempt characteristics.

RESULTS

Spontaneous quit attempts were associated with greater odds of quit success (OR = 1.31, 95% CI = 1.07 to 1.60) but the association was not significant in the fully adjusted model (ORadj = 1.19, 95% CI = 0.95 to 1.49). In this adjusted model, those who attempted to quit without cutting down first (ORadj = 3.08, 95% CI = 2.46 to 3.88) and were male (ORadj = 1.44, 95% CI = 1.16 to 1.80) had greater odds of success; although a greater number of attempts in the past 6 months, stronger urges to smoke (strong vs. none), higher daily cigarette consumption, and lower social grade (E vs. AB) were associated with lower odds of success (ORadj range = 0.32-0.98, p < .030). Quit attempts made without cutting down first were correlated with spontaneous quit attempts (r = .150, p < .001) and appeared to account for the diminished association between spontaneous quitting and success (ORadj = 1.18, 95% CI = 0.96 to 1.46).

CONCLUSIONS

The increased success rate of spontaneous quit attempts appears to be because spontaneous quit attempts are more likely to be made without cutting down first.

IMPLICATIONS

The apparent benefit of spontaneous over planned quit attempts may be attributable to the former being more likely to involve quitting without cutting down first (ie, abrupt cessation) than cutting down first (ie, gradual cessation) and so this may be a more useful target for advice to improve the chances of successful quitting.

Decrease in Resting Heart Rate Measured Using Smartphone Apps to Verify Abstinence From Smoking: An Exploratory Study

INTRODUCTION

Verifying self-reports of smoking abstinence is challenging in studies that involve remote data collection. Resting heart rate (HR) decreases during smoking abstinence. This study assessed whether a decrease in resting HR measured using freely available smartphone apps could potentially be used to verify smoking abstinence.

METHODS

This study involved a repeated measures experimental design, with data collection in natural setting. Participants were 18 adult, daily smokers. They recorded resting HR in beats per minute (bpm) using freely available smartphone apps during five timepoints (two in the morning and three postnoon) on each of 3 days. The outcome measure was the mean of the postnoon HR recordings. The experimental condition for each of the 3 days (counterbalanced order) was as follows: (1) smoking as usual, (2) not smoking without nicotine replacement therapy (NRT), or (3) not smoking but using NRT. Abstinence was verified using expired-air carbon monoxide (CO) concentration.

RESULTS

Compared with the smoking as usual condition, mean HR was 13.4 bpm lower (95% confidence interval [CI] = 5.4 to 21.4, p = .001) in the not smoking without NRT condition and 10.4 bpm lower (95% CI = 3.1 to 17.8, p = 0.004) in the not smoking with NRT condition. There was no statistically significant difference in HR between the two not smoking conditions (p = .39). Abstinence during not smoking days without and with NRT was CO-verified in 18/18 and in 16/18 cases, respectively.

CONCLUSIONS

Self-recording of resting HR in natural setting using smartphone apps shows a reliable decrease in response to smoking abstinence and may provide a basis for remote verification in smoking cessation studies.

IMPLICATIONS

Remote verification of self-reported abstinence in smoking cessation studies remains challenging. Smoking abstinence has been shown to decrease resting HR under laboratory conditions. This study demonstrated that self-recording using freely available smartphone apps shows reliable decreases in resting HR during smoking abstinence and may provide a basis for inexpensive remote verification of smoking abstinence.

Effectiveness of an innovative and interactive smoking cessation training module for dental students: A prospective study

BACKGROUND

Smoking is one of the world's major health problems and dental professionals are in a unique position to promote smoking cessation. However, according to the current literature, neither dental students nor dentists feel adequately prepared to counsel smokers. The purpose of this study was to develop and implement a teaching intervention on smoking cessation for fourth-year dental students and assess its effectiveness in terms of learning outcome on knowledge, communication skills and attitudes.

MATERIALS AND METHODS

In this prospective intervention study, students in the intervention group (n = 28) participated in a teaching module consisting of a podcast, an interactive lecture, a seminar, and small-group sessions with role-play interactions. Knowledge, communication skills and attitudes were measured using written examinations and an objective structured clinical examination (OSCE) at the end of the module and 6 months later. Results were compared with data from a historical control group (n = 27) receiving standard teaching.

RESULTS

Compared with the control group, students in the intervention group had higher scores in the knowledge test (67.1% vs 41.8%; P < .001; d = 2.8) as well as in the OSCE (74.9% vs 44.7%; P < .001; d = 2.3) and also retained more knowledge (52.7% vs 36.5%; P < .001; d = 2.0) and skills (71.8% vs 47.6%; P < .001; d = 2.5) over a period of 6 months. Attitudes were similar across groups and time-points.

CONCLUSION

The teaching intervention equipped dental students with specific knowledge and skills required to effectively counsel smoking patients. Further research is required to assess the transfer of these skills to the clinical setting.

Cross-sectional associations between angiotensin-converting enzyme inhibitor use and cancer diagnosis in US adults

The objective of the present study was to investigate the association between angiotensin-converting enzyme (ACE) inhibitor use and cancer incidence (overall, and breast, prostate, and colorectal cancers specifically) in a large representative sample of US adults. Cross-sectional data on cancer diagnosis, timing of cancer diagnosis, ACE inhibitor use, and other characteristics were extracted from 49 512 adults aged ≥ 20 years participating in the National Health and Nutrition Examination Survey (1999-2016). Multivariable-logistic and propensity score matching (PSM) regressions examined the relationship between pre-diagnosis use of ACE inhibitors and diagnosis of all cancers, and breast, prostate, and colorectal cancers specifically. Overall, we observed an increased likelihood of cancer diagnosis [odds ratio (OR) 1.269, 95% confidence interval (CI) 1.088-1.480] among those who used ACE inhibitors compared to non-ACE inhibitor use, and for prostate cancer diagnosis (OR 1.438, 95% CI 1.090-1.897), after adjusting for age, sex, body mass index, race/ethnicity, educational attainment, physical activity, alcohol drinking status, smoking status, and high blood pressure. PSM regression retrieved more conservative estimates such that the increased likelihood of cancer diagnosis was only observed when comparing ACE inhibitor users with non-drug users (OR 1.022, 95% CI 1.016-1.027). Compared with non-ACE inhibitor use, ACE inhibitor use was associated with an increased risk of prostate cancer. In conclusion, in this large representative sample of US adults, it was found that ACE inhibitor use may have a marginal influence on some cancers.

Pragmatic randomised trial of a smartphone app (NRT2Quit) to improve effectiveness of nicotine replacement therapy in a quit attempt by improving medication adherence: results of a prematurely terminated study

BACKGROUND

Nicotine replacement therapy (NRT) bought over the counter (OTC) appears to be largely ineffective for smoking cessation, which may be partially explained by poor adherence. We developed and evaluated the NRT2Quit smartphone app (for iOS) designed to improve quit attempts with OTC NRT by improving adherence to the medications.

METHODS

This study was a pragmatic double-blind randomised controlled trial with remote recruitment through leaflets distributed to over 300 UK-based community pharmacies. The study recruited adult daily smokers (≥10 cigarettes per day) who bought NRT, wanted to quit smoking, downloaded NTR2Quit and completed the registration process within the app. Participants were automatically randomly assigned within the app to the intervention (full) version of NRT2Quit or to its control (minimal) versions. The primary outcome was biochemically verified 4-week abstinence assessed at 8-week follow-up using Russell Standard criteria and intention to treat. Bayes factors were calculated for the cessation outcome. Secondary outcomes were self-reported abstinence, NRT use, app use and satisfaction with the app.

RESULTS

The study under-recruited. Only 41 participants (3.5% of the target sample) were randomly assigned to NRT2Quit (n = 16) or the control (n = 25) app versions between March 2015 and September 2016. The follow-up rate was 51.2%. The intervention participants had numerically higher biochemically verified quit rates (25.0% versus 8.0%, P = 0.19, odds ratio = 3.83, 0.61-24.02). The calculated Bayes factor, 1.92, showed that the data were insensitive to test for the hypothesis that the intervention app version aided cessation. The intervention participants had higher median logins (2.5 versus 0, P = 0.01) and were more likely to use NRT at follow-up (100.0% versus 28.6%, P = 0.03) and recommend NRT2Quit to others (100.0% versus 28.6%, P = 0.01).

CONCLUSIONS

Despite very low recruitment, there was preliminary but inconclusive evidence that NRT2Quit may improve short-term abstinence and adherence among smokers using NRT. Well-powered studies on NRT2Quit are needed, but different recruitment methods will be required to engage smokers through community pharmacies or other channels.

TRIAL REGISTRATION

ISRCTN ISRCTN33423896 , prospectively registered on 22 March 2015.

Effectiveness of training general practitioners to improve the implementation of brief stop-smoking advice in German primary care: study protocol of a pragmatic, 2-arm cluster randomised controlled trial (the ABCII trial)

Background

The German clinical guideline on tobacco addiction recommends that general practitioners (GPs) provide brief stop-smoking advice to their patients according to the “5A” or the much briefer “ABC” method, but its implementation is insufficient. A lack of training is one barrier for GPs to provide such advice. Moreover, the respective effectiveness of a 5A or ABC training regarding subsequent delivery of stop-smoking advice has not been investigated. We developed a training for GPs according to both methods, and conducted a pilot study with process evaluation to optimize the trainings according to the needs of GPs. This study aims at evaluating the effectiveness of both trainings.

Methods

A pragmatic 2-arm cluster randomised controlled trial with a pre-post data collection will be conducted in 48 GP practices in North Rhine-Westphalia (Germany). GPs will be randomised to receive a 3.5-h-training in delivering either 5A or ABC, including peer coaching and intensive role plays with professional actors. The patient-reported primary outcome (receipt of GP advice to quit: yes/no) and secondary outcomes (recommendation rates of smoking cessation treatments, group comparison (5A versus ABC): receipt of GP advice to quit) will be collected in smoking patients routinely consulting their GP within 4 weeks prior, and 4 weeks following the training. Additional secondary outcomes will be collected at 4, 12 and 26 weeks following the consultation: use of cessation treatments during the last quit attempt (if so) since the GP consultation, and point-prevalence abstinence rates. The primary data analysis will be conducted using a mixed-effects logistic regression model with random effects for the cluster variable.

Discussion

If the training increases the rates of delivery of stop-smoking advice, it would offer a low-threshold strategy for the guideline implementation in German primary care. Should one method prove superior, a more specific guideline recommendation can be proposed.

Trial registration

German Clinical Trials Register (DRKS00012786); registered on 22th August 2017, prior to the first patient in.

Electronic supplementary material

The online version of this article (10.1186/s12875-019-0986-8) contains supplementary material, which is available to authorized users.

The Impact of Medical Students' Individual Teaching Format Choice on the Learning Outcome Related to Clinical Reasoning

BACKGROUND

Repeated formative assessments using key feature questions have been shown to enhance clinical reasoning. Key feature questions augmented by videos presenting clinical vignettes may be more effective than text-based questions, especially in a setting where medical students are free to choose the format they would like to work with. This study investigated learning outcomes related to clinical reasoning in students using video- or text-based key feature questions according to their individual preferences.

OBJECTIVE

The aim of this study was to test the hypothesis that repeated exposure to video-based key feature questions enhances clinical reasoning to a greater extent than repeated exposure to text-based key feature questions if students are allowed to choose between those different formats on their own.

METHODS

In this monocentric, prospective, nonrandomized trial, fourth-year medical students attended 12 computer-based case seminars during which they worked on case histories containing key feature questions. Cases were available in a text- and a video-based format. Students chose their preferred presentation format at the beginning of each case seminar. Student performance in key feature questions was assessed in formative entry, exit, and retention exams and was analyzed with regard to preceding exposure to video- or text-based case histories.

RESULTS

Of 102 eligible students, 75 provided written consent and complete data at all study exams (response rate=73.5%). A majority of students (n=52) predominantly chose the text-based format. Compared with these, students preferring the video-based format achieved a nonsignificantly higher score in the exit exam (mean 76.2% [SD 12.6] vs 70.0% [SD 19.0]; P=.15) and a significantly higher score in the retention exam (mean 75.3% [SD 16.6] vs 63.4% [SD 20.3]; P=.02). The effect was independent of the video- or text-based presentation format, which was set as default in the respective exams.

CONCLUSIONS

Despite students' overall preference for text-based case histories, the learning outcome with regard to clinical reasoning was higher in students with higher exposure to video-based items. Time-on-task is one conceivable explanation for these effects as working with video-based items was more time-consuming. The baseline performance levels of students do not account for the results as the preceding summative exam results were comparable across the 2 groups. Given that a substantial number of students chose a presentation format that was less effective, students might need to be briefed about the beneficial effects of using video-based case histories to be able to make informed choices about their study methods.

National survey of evaluation practices and performance-guided resource allocation at German medical schools

Background: Little is known about evaluation practices as well as performance-oriented allocation of resources according to teaching quality at German medical schools. For this reason, the Association of the Scientific Medical Societies in Germany and the German Association of Medical Faculties aimed to analyse current practices at German medical schools. Methods: Data were collected by a questionnaire which was sent to all medical schools in Germany. Results: 30 medical schools with 33 undergraduate medical programs participated in the survey (response rate: 83%). The evaluation tools used at these schools mainly assessed structural and procedural aspects of teaching and were designed to obtain overall student ratings of teaching quality. Evaluation tools were quite heterogeneous across the sample, and some uncertainty remained with regard to the psychometric properties of these tools and whether they meet international quality standards. Various algorithms underlying resource allocation for teaching are being used, but most focus on quantity rather than quality of teaching. Conclusion: A nationwide agreement on a generalizable definition of high-quality teaching is desirable. At the same time, reliable and valid tools measuring teaching quality need to be identified and/or created. This could be accomplished through a wider collaboration of medical schools and could represent an advancement for the allocation of resources for high-quality teaching.

Serious Games in Surgical Medical Education: A Virtual Emergency Department as a Tool for Teaching Clinical Reasoning to Medical Students

Background

Serious games enable the simulation of daily working practices and constitute a potential tool for teaching both declarative and procedural knowledge. The availability of educational serious games offering a high-fidelity, three-dimensional environment in combination with profound medical background is limited, and most published studies have assessed student satisfaction rather than learning outcome as a function of game use.

Objective

This study aimed to test the effect of a serious game simulating an emergency department (“EMERGE”) on students’ declarative and procedural knowledge, as well as their satisfaction with the serious game.

Methods

This nonrandomized trial was performed at the Department of General, Visceral and Cancer Surgery at University Hospital Cologne, Germany. A total of 140 medical students in the clinical part of their training (5th to 12th semester) self-selected to participate in this experimental study. Declarative knowledge (measured with 20 multiple choice questions) and procedural knowledge (measured with written questions derived from an Objective Structured Clinical Examination station) were assessed before and after working with EMERGE. Students’ impression of the effectiveness and applicability of EMERGE were measured on a 6-point Likert scale.

Results

A pretest-posttest comparison yielded a significant increase in declarative knowledge. The percentage of correct answers to multiple choice questions increased from before (mean 60.4, SD 16.6) to after (mean 76.0, SD 11.6) playing EMERGE (P<.001). The effect on declarative knowledge was larger in students in lower semesters than in students in higher semesters (P<.001). Additionally, students’ overall impression of EMERGE was positive.

Conclusions

Students self-selecting to use a serious game in addition to formal teaching gain declarative and procedural knowledge.

Quality Principles of App Description Texts and Their Significance in Deciding to Use Health Apps as Assessed by Medical Students: Survey Study

Background

Currently, there are no binding requirements for manufacturers prescribing which information must be included in the app descriptions of health apps.

Objective

The aim of this study was to investigate how medical students perceive a selection of quality principles, intended for usage decisions in the app context, and establish whether the information presented in a sample of app descriptions is perceived as sufficient for facilitating an informed usage decision.

Methods

A total of 123 students (mean age 24.2 years, SD 3.4) participating in a 6-week teaching module covering cardiology and pulmonology at the University of Göttingen (original enrollment 152 students, response rate 80.9%) were included. Students were asked to read 3 store description texts of cardiological or pneumological apps and initially assess whether the descriptions sufficed for a usage decision. Subsequently, they were queried on their perception of the relevance of 9 predefined quality principles, formulated for usage decisions. An appraisal of whether the app description texts contained sufficient information to satisfy these quality principles followed. By means of 20 guiding questions, participants were then asked to identify relevant information (or a lack thereof) within the descriptions. A reassessment of whether the description texts sufficed for making a usage decision ensued. A total of 343 complete datasets were obtained.

Results

A majority of the quality principles were described as “very important” and “important” for making a usage decision. When accessed via the predefined principles, students felt unable to identify sufficient information within the app descriptions in 68.81% (2124/3087) of cases. Notably, information regarding undesired effects (91.8%, 315/343), ethical soundness (90.1%, 309/343), measures taken to avert risks (89.2%, 306/343), conflicts of interest (88.3%, 303/343), and the location of data storage (87.8%, 301/343) was lacking. Following participants’ engagement with the quality principles, statistically significant changes in their assessment of whether the app descriptions sufficed for a usage decision can be seen—McNemar-Bowker test (3)=45.803919, P<.001, Cohen g=.295. In 34.1% (117/343) cases, the assessment was revised. About 3 quarters of changed assessments were seen more critically (76.9%, 90/117). Although, initially, 70% (240/343) had been considered “sufficient,” this rate was reduced to 54.2% (186/343) in the second assessment.

Conclusions

In a considerable number of app descriptions, participants were unable to locate the information necessary for making an informed usage decision. Participants’ sensitization to the quality principles led to changes in their assessment of app descriptions as a tool for usage decisions. Better transparency in app descriptions released by manufacturers and the exposure of users to quality principles could collectively form the basis for well-founded usage decisions.

Are smokers who are regularly exposed to e-cigarette use by others more or less motivated to stop or to make a quit attempt? A cross-sectional and longitudinal survey

BACKGROUND

Concerns have been raised that observing other people using e-cigarettes may undermine motivation to quit by renormalising smoking. This study aimed to explore associations between regular exposure to other people's e-cigarette use and motivation to stop smoking and quit attempts in smokers.

METHODS

Data were from 12,787 smokers in England who participated in the Smoking Toolkit Study between November 2014 and May 2018. At baseline, respondents were asked whether anyone other than themselves regularly used an e-cigarette in their presence, whether they had made a quit attempt in the past year and how motivated they were to stop. Data at 6-month follow-up were available for 1580 respondents, who reported on whether they had attempted to quit in the past 6 months.

RESULTS

Smokers who reported regular exposure to e-cigarette use by others were more likely than those who did not to have tried to stop smoking in the past year (32.3% vs. 26.8%; unadjusted RR 1.21, 95% CI 1.11-1.31) and have high motivation to quit (16.6% vs. 14.2%; unadjusted RR 1.17, 95% CI 1.05-1.30) but were not significantly more or less likely to make a quit attempt over the subsequent 6 months (34.4% vs. 31.3%; unadjusted RR 1.10, 95% CI 0.88-1.38). In models that adjusted for participants' own current e-cigarette use and unadjusted and adjusted models excluding current e-cigarette users from the sample, there were no significant associations between exposure to e-cigarette use by others and past quit attempts (RR 0.97-1.00), high current motivation to quit (RR 0.97-1.00) or prospective quit attempts (RR 0.94-1.12). In contrast, exposure to use of cigarettes was associated with low motivation to quit even after adjustment (RR 0.89) but not with quit attempts. Participants' own use of e-cigarette was strongly associated with high motivation to quit (RR 1.95) and past quit attempts (RR 2.14) and appeared to account for the bivariate associations with reported exposure to e-cigarettes.

CONCLUSION

Smokers who report regular exposure to other people using e-cigarettes are more likely to report past quit attempts and high current motivation to quit, but there does not appear to be an independent association with motivation or quit attempts after adjustment for their own current use of e-cigarettes. In contrast, reported exposure to other people using cigarettes was independently and negatively associated with high motivation.

Effect of different scoring approaches upon credit assignment when using Multiple True-False items in dental undergraduate examinations

INTRODUCTION

Various scoring approaches for Multiple True-False (MTF) items exist. This study aimed at comparing scoring results obtained with different scoring approaches and to assess the effect of item cues on each scoring approaches' result.

MATERIALS AND METHODS

Different scoring approaches (MTF, Count-2, Count-3, "Vorkauf-Method," PS₅₀ , Dichotomized MTF, "Blasberg-Method," Multiple response (MR), Correction for Guessing, "Ripkey-Method," Morgan-Method, Balanced Scoring Method) were retrospectively applied to all MTF items used within electronic examinations of undergraduate dental students at the University Medical Center Göttingen in the winter term 2016/2017 (1297 marking events). Item quality was evaluated regarding formal parameters such as presence of cues and correctness of content. Differences between scoring results of all scoring approaches and the differences between each methods' scoring results of items with and without cues were calculated by Wilcoxon rank sum tests (P < .05).

RESULTS

Average scoring results per item highly differed between the scoring approaches and ranged from 0.46 (MR) to 0.92 (Dichotomized MTF). Presence of cues leads to significantly higher scoring in case of all scoring approaches (P < .001; +0.14 on average). However, effect of cues differed amongst scoring approaches and ranged from +0.04 (Dichotomized MTF) to +0.20 (MR).

CONCLUSION

Scoring of MTF items is complex. The data presented in this manuscript may help educators make informed choices about scoring algorithms.

Training of clinical reasoning with a Serious Game versus small-group problem-based learning: A prospective study

INTRODUCTION

Serious Games are increasingly being used in undergraduate medical education. They are usually intended to enhance learning with a focus on knowledge acquisition and skills development. According to the current literature, few studies have assessed their effectiveness regarding clinical reasoning (CR). The aim of this prospective study was to compare a Serious Game, the virtual Accident & Emergency department 'EMERGE' to small-group problem-based learning (PBL) regarding student learning outcome on clinical reasoning in the short term.

METHODS

A total of 112 final-year medical students self-selected to participate in ten 90-minute sessions of either small-group PBL or playing EMERGE. CR was assessed in a formative examination consisting of six key feature cases and a final 45-minute EMERGE session.

RESULTS

Overall, the EMERGE group (n = 78) scored significantly higher than the PBL group (n = 34) in the key feature examination (62.5 (IQR: 17.7)% vs. 54.2 (IQR: 21.9)%; p = 0.015). There was no significant difference in performance levels between groups regarding those cases which had been discussed in both instructional formats during the training phase. In the final EMERGE session, the EMERGE group achieved significantly better results than the PBL group in all four cases regarding the total score as well as in three of four cases regarding the final diagnosis and the correct therapeutic interventions.

CONCLUSION

EMERGE can be used effectively for CR training in undergraduate medical education. The difference in key feature exam scores was driven by additional exposure to more cases in EMERGE compared to PBL despite identical learning time in both instructional formats. EMERGE is a potential alternative to intensive small-group teaching. Further work is needed to establish how Serious Games enhance CR most effectively.

How can we teach medical students to choose wisely? A randomised controlled cross-over study of video- versus text-based case scenarios

BACKGROUND

The Choosing Wisely campaign highlights the importance of clinical reasoning abilities for competent and reflective physicians. The principles of this campaign should be addressed in undergraduate medical education. Recent research suggests that answering questions on important steps in patient management promotes knowledge retention. It is less clear whether increasing the authenticity of educational material by the inclusion of videos further enhances learning outcome.

METHODS

In a prospective randomised controlled cross-over study, we assessed whether repeated video-based testing is more effective than repeated text-based testing in training students to choose appropriate diagnostic tests, arrive at correct diagnoses and identify advisable therapies. Following an entry exam, fourth-year undergraduate medical students attended 10 weekly computer-based seminars during which they studied patient case histories. Each case contained five key feature questions (items) on the diagnosis and treatment of the presented patient. Students were randomly allocated to read text cases (control condition) or watch videos (intervention), and assignment to either text or video was switched between groups every week. Using a within-subjects design, student performance on video-based and text-based items was assessed 13 weeks (exit exam) and 9 months (retention test) after the first day of term. The primary outcome was the within-subject difference in performance on video-based and text-based items in the exit exam.

RESULTS

Of 125 eligible students, 93 provided data for all three exams (response rate 74.4%). Percent scores were significantly higher for video-based than for text-based items in the exit exam (76.2 ± 19.4% vs. 72.4 ± 19.1%, p = 0.026) but not the retention test (69.2 ± 20.2% vs. 66.4 ± 20.3%, p = 0.108). An additional Bayesian analysis of this retention test suggested that video-based training is marginally more effective than text-based training in the long term (Bayes factor 2.36). Regardless of presentation format, student responses revealed a high prevalence of erroneous beliefs that, if applied to the clinical context, could place patients at risk.

CONCLUSION

Repeated video-based key feature testing produces superior short-term learning outcome compared to text-based testing. Given the high prevalence of misconceptions, efforts to improve clinical reasoning training in medical education are warranted. The Choosing Wisely campaign lends itself to being part of this process.

The effect of repeated testing vs repeated practice on skills learning in undergraduate dental education

OBJECTIVES

Recent studies in undergraduate medical education have demonstrated the advantage of repeated testing over repeated practice with regard to knowledge and skills retention. The aim of this study was to investigate whether this "testing effect" also applies to skills retention in undergraduate dental education.

METHODS

In this prospective, randomised controlled trial, fourth-year dental students at Göttingen University Medical Centre participated in a training session on surgical suturing in winter term 2014/2015. Following this, they were either assigned to two sessions of additional skills training (group A) or two sessions of skills assessment with feedback (group B). These sessions were spaced over a period of 4 weeks. Skills retention was assessed in a summative objective structured clinical examination (OSCE) at the end of term, that is 6 months after the initial teaching session.

RESULTS

A total of 32 students completed the study. With regard to suturing, OSCE performance was significantly better in group B than group A (81.9±13.1% vs 63.0±15.4%; P=0.001; Cohen's d=1.33). There was no significant OSCE performance difference in the two groups with regard to other learning objectives that were addressed in the end-of-term examination. Thus, the group difference was specific to suturing skills.

CONCLUSIONS

This is the first study to demonstrate that in dental education, repeated testing produces more favourable skills retention than repeated practice. Test-enhanced learning might be a viable concept for skills retention in undergraduate dentistry education.

[Choosing Wisely in medical education]

Choosing Wisely recommendations address situations where physicians will have to make decisions about further diagnostic and therapeutic steps. Undergraduate medical education needs to equip students with the foundations on which clinical reasoning skills can be acquired and fostered throughout their clinical career. Teaching these skills usually involves patients (e.g., bedside teaching, electives, clinical attachments) but it can also be delivered in the format of formalised small-group, case-based learning. Case-based key feature tests have been developed to facilitate the assessment of learning outcomes related to clinical reasoning. Repeated testing with key feature cases yields better medium-term retention than repeatedly studying the same material (without questions). The project 'Choosing Wisely in medical education', which was funded by the German Association for Internal Medicine, involves the creation of key feature cases with reference to the German set of Choosing Wisely recommendations. This article presents the results of the first pilot study using these new cases.

Facilitators of high-quality teaching in medical school: findings from a nation-wide survey among clinical teachers

BACKGROUND

Clinical teachers in medical schools are faced with the challenging task of delivering high-quality patient care, producing high-impact research and contributing to undergraduate medical education all at the same time. Little is known on the gap between an 'ideal' environment supporting clinical teachers to provide high quality teaching for their students and the reality of clinical teaching during worktime in the clinical environment. Most quantitative research published so far was done in a wide range of medical educators and did not consider individual academic qualifications. In this study, we wanted to survey clinical teachers in particular and assess the potential impact of individual academic qualification on their perceptions.

METHODS

Based on qualitative data of focus group discussions, we developed a questionnaire which was piloted among 189 clinical teachers. The final web-based questionnaire was completed by clinical teachers at nine German medical schools.

RESULTS

A total of 833 clinical teachers (569 junior physicians, 264 assistant professors) participated in the online survey. According to participants, the most important indicator of high quality teaching was "sustained student learning outcome" followed by "stimulation of interest in the subject matter". Lack of time was the main factor impeding effective teaching (78%). Among the factors facilitating high-quality teaching, protected preparation time during working hours (48%) and more recognition of high-quality teaching within medical schools (21%) were perceived as most helpful. Three out of four teachers (76%) were interested in faculty development programmes directed at teaching skills, but 60% stated they had no time to engage in such activities. With regard to evaluation, teachers preferred individual feedback (75%) over global ratings (21%). Differences between assistant professors and junior physicians were found in that the latter group perceived their teaching conditions as more difficult.

CONCLUSIONS

Lack of time is a major barrier against planning and delivering good clinical teaching in medical schools. According to our findings, the situation at German medical schools is particularly challenging for junior physicians. Creating an institutional culture in which teaching is regarded as highly as patient care and research is a prerequisite for overcoming the barriers identified in this study.

Evaluation of large-group lectures in medicine - development of the SETMED-L (Student Evaluation of Teaching in MEDical Lectures) questionnaire

BACKGROUND

The seven categories of the Stanford Faculty Development Program (SFDP) represent a framework for planning and assessing medical teaching. Nevertheless, so far there is no specific evaluation tool for large-group lectures that is based on these categories. This paper reports the development and psychometric validation of a short German evaluation tool for large-group lectures in medical education (SETMED-L: 'Student Evaluation of Teaching in MEDical Lectures') based on the SFDP-categories.

METHODS

Data were collected at two German medical schools. In Study 1, a full information factor analysis of the new 14-item questionnaire was performed. In Study 2, following cognitive debriefings and adjustments, a confirmatory factor analysis was performed. The model was tested for invariance across medical schools and student gender. Convergent validity was assessed by comparison with results of the FEVOR questionnaire.

RESULTS

Study 1 (n = 922) yielded a three-factor solution with one major (10 items) and two minor factors (2 items each). In Study 2 (n = 2740), this factor structure was confirmed. Scale reliability ranged between α = 0.71 and α = 0.88. Measurement invariance was given across student gender but not across medical schools. Convergent validity in the subsample tested (n = 246) yielded acceptable results.

CONCLUSION

The SETMED-L showed satisfactory to very good psychometric characteristics. The main advantages are its short yet comprehensive form, the integration of SFDP-categories and its focus on medical education.