ChatGPT in medical school: how successful is AI in progress testing?

BACKGROUND

As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test.

METHODS

A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question.

RESULTS

Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08, 95% CI [-0.18, 0.02], t(393) = -1.55, p = 0.121; for word count rho = -0.03, 95% CI [-0.13, 0.07], t(393) = -0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002).

CONCLUSION

ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.

Mixed Reality as a Teaching Tool for Medical Students in Neurosurgery

Background and Objectives: Simulation-based learning within neurosurgery provides valuable and realistic educational experiences in a safe environment, enhancing the current teaching model. Mixed reality (MR) simulation can deliver a highly immersive experience through head-mounted displays and has become one of the most promising teaching tools in medical education. We aimed to identify whether an MR neurosurgical simulation module within the setting of an undergraduate neurosurgical hands-on course could improve the satisfaction of medical students. Materials and Methods: The quasi-experimental study with 223 medical students [120 in the conventional group (CG) and 103 in the MR-group (MRG)] was conducted at the University Hospital Münster, Münster, Germany. An MR simulation module was presented to the intervention group during an undergraduate neurosurgical hands-on course. Images of a skull fracture were reconstructed into 3D formats compatible with the MR-Viewer (Brainlab, Munich, Germany). Participants could interact virtually with the model and plan a surgical strategy using Magic Leap goggles. The experience was assessed by rating the course on a visual analog scale ranging from 1 (very poor) to 100 (very good) and an additional Likert-scale questionnaire. Results: The satisfaction score for CG and MRG were 89.3 ± 13.3 and 94.2 ± 7.5, respectively. The Wilcoxon rank-sum test showed that MR users (Mdn = 97.0, IQR = 4, n = 103) were significantly more satisfied than CG users (Mdn = 93.0, IQR = 10, n = 120; ln(W) = 8.99, p < 0.001) with moderate effect size (r^biserial = 0.30, CI95 [0.15, 0.43]), thus indicating that the utilization of MR-simulation is associated with greater satisfaction. Conclusions: This study reports a positive response from medical students towards MR as an educational tool. Feedback from the medical students encourages the adoption of disruptive technologies into medical school curricula.

Clinical and imaging factors that can predict contagiousness of pulmonary tuberculosis

BACKGROUND

Knowledge on predicting pulmonary tuberculosis (PTB) contagiosity in the hospital admission setting is limited. The objective was to assess clinical and radiological criteria to predict PTB contagiosity.

METHODS

Retrospective analysis of 7 clinical, 4 chest X-ray (CXR) and 5 computed tomography (CT) signs in 299 PTB patients admitted to an urban tertiary hospital from 2008 to 2016. If the acid fact bacilli stain was positive (AFB+) on admission, the case was considered high contagiosity.

RESULTS

Best predictors for high PTB contagiosity (AFB+) were haemoptysis (OR 4.33), cough (3.00), weight loss (2.96), cavitation in CT (2.75), cavitation in CXR (2.55), tree-in-bud-sign in CT (2.12), German residency of the patient (1.89), and abnormal auscultation findings (1.83). A previous TB infection reduced the risk of contagiosity statistically (0.40). Radiographic infiltrates, miliary picture, and pleural effusion were not helpful in predicting high or low contagiosity. 34% of all patients were clinically asymptomatic (20% of the highly contagious group, 50% of the low contagious group).

CONCLUSION

Haemoptysis, cough and weight loss as well as cavitation and tree-in-bud sign in CXR/CT can be helpful to predict PTB contagiosity and to improve PTB management.

Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education - a retrospective, single-center, mixed model analysis of progress testing results

Medical education research focuses on the development of efficient learning methods promoting the acquisition of student's knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional 'don't know' option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1^st and 2^nd, d = 0.88 (total gain: 12.0%) between the 2^nd and 3^rd, d = 0.60 (total gain: 7.9%) between the 3^rd and 4^th and d = 0.58 (total gain: 7.9%) between the 4^th and 5^th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53-0.9 by study year may help researchers to design studies in medical education.

Risk literacy assessment of general practitioners and medical students using the Berlin Numeracy Test

Background

The responsibility for helping patients understand potential health benefits and risks, especially regarding screening tests, falls largely to general practitioners (GPs). The Berlin Numeracy Test (BNT) specifically measures risk literacy (i.e., the ability to understand different aspects of statistical numeracy associated with accurate interpretation of information about risks). This study explored the association between risk literacy levels and clinical experience in GPs vs. medical students. Additionally, the effect of GP risk literacy on evaluation of the predictive value of screening tests was examined.

Methods

The participants were 84 GPs and 92 third-year medical students who completed the BNT (total score range 0–4 points). The GPs received an additional case scenario on mammography screening as a simple measure of performance in applying numeracy skills.

Results

Despite having an average of 25.9 years of clinical experience, GPs scored no better than medical students on risk literacy (GPs: 2.33 points, 95% confidence interval [CI] 2.08–2.59; students: 2.34, 95% CI 2.07–2.61; P = .983). Of all GPs, 71.6% (n = 58) greatly overestimated the real predictive value.

Conclusions

In this study, we found no difference in risk literacy between current students and current GPs. GPs lack risk literacy and consequently do not fully understand numeric estimates of probability in routine screening procedures.

Combination of problem-based learning with high-fidelity simulation in CPR training improves short and long-term CPR skills: a randomised single blinded trial

BACKGROUND

Performance of sufficient cardiopulmonary resuscitation (CPR) by medical personnel is critical to improve outcomes during cardiac arrest. It has however been shown that even health care professionals possess a lack of knowledge and skills in CPR performance. The optimal method for teaching CPR remains unclear, and data that compares traditional CPR instructional methods with newer modalities of CPR instruction are needed. We therefore conducted a single blinded, randomised study involving medical students in order to evaluate the short- and long-term effects of a classical CPR education compared with a bilateral approach to CPR training, consisting of problem-based learning (PBL) plus high fidelity simulation.

METHODS

One hundred twelve medical students were randomized during a curricular anaesthesiology course to a control (n = 54) and an intervention (n = 58) group. All participants were blinded to group assignment and partook in a 30-min-lecture on CPR basics. Subsequently, the control group participated in a 90-min tutor-guided CPR hands-on-training. The intervention group took part in a 45-min theoretical PBL module followed by 45 min of high fidelity simulated CPR training. The rate of participants recognizing clinical cardiac arrest followed by sufficiently performed CPR was the primary outcome parameter of this study. CPR performance was evaluated after the intervention. In addition, a follow-up evaluation was conducted after 6 months.

RESULTS

51.9% of the intervention group met the criteria of sufficiently performed CPR as compared to only 12.5% in the control group on the day of the intervention (p = 0.007). Hands-off-time as a marker for CPR continuity was significantly less in the intervention group (24.0%) as compared to the control group (28.3%, p = 0.007, Hedges' g = 1.55). At the six-month follow-up, hands-off-time was still significantly lower in the intervention group (23.7% vs. control group: 31.0%, p = 0.006, Hedges' g = 1.88) but no significant difference in sufficiently performed CPR was detected (intervention group: 71.4% vs. control group: 54.5%, p = 0.55).

CONCLUSION

PBL combined with high fidelity simulation training leads to a measurable short-term increase in initiating sufficient CPR by medical students immediately after training as compared to classical education. At six month post instruction, these differences remained only partially.

Simulation-based mastery learning in medical students: Skill retention at 1-year follow up

Context: Deficits in basic skill performance and long-term skill retention among medical students and novice doctors are a persistent problem. This controlled study tested whether the addition of a mastery learning component to simulation-based teaching is associated with long-term retention and performance of peripheral venous catheter insertion. Methods: Fourth-year medical students were assigned to receive either the control (simulation without mastery learning, n = 131) or the intervention (simulation + mastery learning, n = 133) instruction in peripheral venous catheter insertion. Performance was assessed at one year post-instruction. Eighty-four students from the control group and 71 from the intervention group participated in the assessment. Results: Students who received the mastery learning instruction achieved higher overall test scores than did controls (median mastery learning score: 20.0, IQR 2.0; median control score 19.0, IQR 3.0; Mann-Whitney U test, p < 0.001, effect size d = 0.82). Pass rates also differed significantly between the groups, with 74.5% (n = 53) of the intervention group passing compared with 33% (n = 28) of the control group (p < 0.001). Conclusions: Mastery learning is an effective means of teaching practical skills to medical students, and is associated with higher scores at a 1-year follow up.

Physicians' comprehension of the German Patients' Rights Act: A cross-sectional study

OBJECTIVE

The German Patients' Rights Act (PRA), promulgated on February 25, 2013, was created to enhance transparency of patients' rights. This prospective study aimed to objectively measure physicians' comprehension of the PRA.

METHODS

We generated a controlled study design, developing a questionnaire consisting of six case scenarios with 4-7 dichotomous items each. The survey concluded with seven 5-point-Likert scale questions, dealing with the PRA's effects. Physicians teaching at the Westfälische Wilhelms-Universität (WWU) Münster served as the intervention group, and medical students from WWU Münster at the beginning of their clinical education formed the control group. Physicians were surveyed in November 2015; students were surveyed in February 2016.

RESULTS

A total 56 completed surveys of physicians and 134 of students were analyzed. Of a total 33 points, on average physicians answered 21.04 (95% confidence interval (CI) 20.43-21.64) items correctly, a significantly higher result than students' 19.74 (95% CI 19.31-20.17) points (p < 0.001; Hedges' g = 0.53). Estimations of the PRA's effects were ambiguous. Students agreed with the PRA's supporting effect more often than physicians (p < 0.001) whereas physicians felt increased uncertainty arising from the PRA.

CONCLUSION

Comprehension of the PRA increases significantly over the course of medical work experience; however, this comprehension is limited among medical experts. The PRA leads to ambiguity and uncertainty in the medical decision-making process.

High Diagnostic Value of a New Real-Time Pneumocystis PCR from Bronchoalveolar Lavage in a Real-Life Clinical Setting

BACKGROUND

To diagnose Pneumocystis jirovecii pneumonia (PCP), PCR testing in bronchoalveolar lavage (BAL) fluid has recently become an alternative to immunofluorescence testing (IFT); however, its diagnostic accuracy is less clear.

OBJECTIVE

To analyze the diagnostic value of a new semiquantitative real-time PCR (RT-PCR) in BAL in a real-life clinical setting.

METHODS

Retrospective analysis of all RT-PCR results [semiquantitative: negative, weakly positive, and strongly positive; measured in cycle thresholds (Ct)] in BAL in the period between 2010 and 2014. The diagnosis of PCP was defined by clinical, radiological, and laboratory signs and by treatment initiation. Any positive PCR was compared with subsequent IFT.

RESULTS

Of 128 patient samples, 32 had PCP. There is a relevant correlation of high significance between positive PCR Ct and IFT (r = -0.7781, p < 0.001), which amounts to about 60% of the variance. Sensitivity, specificity, and positive predictive values (PPV) of any positive RT-PCR were 100, 80, and 63%, respectively. No patient with negative RT-PCR had PCP. Specificity and PPV are 100% in strongly positive RT-PCR, whereas they decrease to 80 and 21% in weakly positive RT-PCR.

CONCLUSION

A negative RT-PCR (Ct >45) rules out PCP. A strongly positive PCR (Ct <31.5) confirms PCP. In these cases, the diagnostic value of the new method is at least equal to the IFT. A weakly positive PCR probably represents pneumocystis colonization and can occur under PCP treatment.

Mastery learning improves students skills in inserting intravenous access: a pre-post-study

OBJECTIVE

Inserting peripheral venous catheters (PVCs) has been identified as a core competency for medical students. Because the performance - even of hygienic standards - of both students and novice physicians is frequently inadequate, medical faculties must focus on competence-based learning objectives and deliberate practice, features that are combined in mastery learning. Our aim was to determine the competency of students in inserting PVCs before and after an educational intervention.

DESIGN

This study comprised a skills assessment with pre- and post-tests of a group of third-year students who received a simulation-based intervention. A newly established curriculum involved one hour of practice at inserting PVCs on simulators. Students were required to pass a test (total 21 points, pass mark 20 points) developed on the concept of mastery learning. An unannounced follow-up test was performed one week (8 days) after the intervention.

SETTING

The simulation center of the medical faculty in Muenster.

PARTICIPANTS

Third-year students who received the intervention.

RESULTS

One hundred and nine complete data sets were obtained from 133 students (82.5%). Most students (97.2%) passed the test after the intervention (mean score increase from 15.56 to 20.50, P<0.001). There was a significant decrease in students' performance after one week (8 days): only 74.5% of participants passed this retest (mean score reduction from 20.50 to 20.06, P<0.001).

CONCLUSION

Mastery learning is an effective form of teaching practical skills to medical students, allowing a thorough preparation for the challenges of daily clinical practice.

Optimizing the quality of hospital discharge summaries--a systematic review and practical tools

OBJECTIVE

Although doctors' discharge summaries (DS) are important forms of communication between the physicians in patient care, deficits in the quality of DS are common. This review aims to answer the following question: according to the literature, how can the quality of DS be improved by (1) interventions; (2) reviews and guidelines of regulatory bodies; and (3) other practical recommendations?

METHODS

Systematic review of the literature.

RESULTS

The scientific papers on optimizing the quality of DS (n = 234) are heterogeneous and do not allow any meta-analysis. The interventional studies revealed that a structured approach of writing, educational training including feedback and the use of a checklist are effective methods. Guidelines are helpful for outlining the key characteristics of DS. Additionally, the articles in the literature provided practical proposals on improving form, structure, clinical content, treatment recommendations, follow-up plan, medications and changes, addressees, patient data, length, language, dictation, electronic processing and timeliness of DS.

CONCLUSION

The literature review revealed various possibilities for improving the quality of DS.

Teaching the concept of brain death in undergraduate medical education

OBJECTIVE

To establish and evaluate a new approach to teach medical students how to assess brain death in patients.

DESIGN, SETTING, AND PARTICIPANTS

A total of 120 fourth-year medical students at Münster Medical School (Germany) participated in a 1-hour lecture on how to assess brain death in patients. After this lecture, students were assigned to 2 groups. One group attended an additional practical course on the evaluation of brain death and received training using a new high-fidelity simulation device. The other group did not participate in any additional training session. All students completed a questionnaire before the lecture and a second questionnaire at the conclusion of the study. For the group undergoing the additional training, the second questionnaire was completed after the additional training session.

RESULTS

The additional practical training session significantly improved the students' performance in assessing brain death and promoted the self-assessment and motivation of the medical students.

CONCLUSIONS

The establishment of a new practical teaching concept led to significant improvements in medical students' assessments of brain death in a practical session. These improvements in medical education could have significant implications for the clinical assessment of patients in the future.

Practicing evidence based medicine at the bedside: a randomized controlled pilot study in undergraduate medical students assessing the practicality of tablets, smartphones, and computers in clinical life

Background

Practicing evidence-based medicine is an important aspect of providing good medical care. Accessing external information through literature searches on computer-based systems can effectively achieve integration in clinical care. We conducted a pilot study using smartphones, tablets, and stationary computers as search devices at the bedside. The objective was to determine possible differences between the various devices and assess students’ internet use habits.

Methods

In a randomized controlled pilot study, 120 students were divided in three groups. One control group solved clinical problems on a computer and two intervention groups used mobile devices at the bedside. In a questionnaire, students were asked to report their internet use habits as well as their satisfaction with their respective search tool using a 5-point Likert scale.

Results

Of 120 surveys, 94 (78.3%) complete data sets were analyzed. The mobility of the tablet (3.90) and the smartphone (4.39) was seen as a significant advantage over the computer (2.38, p < .001). However, for performing an effective literature search at the bedside, the computer (3.22) was rated superior to both tablet computers (2.13) and smartphones (1.68). No significant differences were detected between tablets and smartphones except satisfaction with screen size (tablet 4.10, smartphone 2.00, p < .001).

Conclusions

Using a mobile device at the bedside to perform an extensive search is not suitable for students who prefer using computers. However, mobility is regarded as a substantial advantage, and therefore future applications might facilitate quick and simple searches at the bedside.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-014-0113-7) contains supplementary material, which is available to authorized users.

Combining simulated patients and simulators: pilot study of hybrid simulation in teaching cardiac auscultation

Auscultation torsos are widely used to teach position-dependent heart sounds and murmurs. To provide a more realistic teaching experience, both whole body auscultation mannequins and torsos have been used in clinical examination skills training at the Medical Faculty of the University of Muenster since the winter term of 2008-2009. This training has since been extended by simulated patients, which are normal, healthy subjects who have undergone attachment of the electronic components of the auscultation mannequins to their chests to mimic pathophysiological conditions ("hybrid models"). The acceptance of this new learning method was examined in the present pilot study. In total, 143 students in their second preclinical year who were participating in auscultation training were randomized into an intervention group (hybrid models) and a control group (auscultation mannequins). One hundred forty-two (99.3%) of these students completed a self-assessment Likert-scale questionnaire regarding different teaching approaches (where 1 = "very poor" to 100 = "very good"). The questionnaire focused on the "value of learning" of different teaching approaches. Direct comparison showed that students evaluated the hybrid models to be significantly more effective than the auscultation mannequins (median: 83 vs. 64, P < 0.001). The cardiac auscultation training was generally assessed positively (median: 88). Additionally, verbal feedback was obtained from simulated patients and tutors (trained students who had successfully passed the course a few semesters earlier). Personal feedback showed high satisfaction from student tutors and simulated patients. Hybrid simulators for teaching cardiac auscultation elucidated positive responses from students, tutors, and simulated patients.

Measuring the ambiguity tolerance of medical students: a cross-sectional study from the first to sixth academic years

BACKGROUND

Tolerance of ambiguity, or the extent to which ambiguous situations are perceived as desirable, is an important component of the attitudes and behaviors of medical students. However, few studies have compared this trait across the years of medical school. General practitioners are considered to have a higher ambiguity tolerance than specialists. We compared ambiguity tolerance between general practitioners and medical students.

METHODS

We designed a cross-sectional study to evaluate the ambiguity tolerance of 622 medical students in the first to sixth academic years. We compared this with the ambiguity tolerance of 30 general practitioners. We used the inventory for measuring ambiguity tolerance (IMA) developed by Reis (1997), which includes three measures of ambiguity tolerance: openness to new experiences, social conflicts, and perception of insoluble problems.

RESULTS

We obtained a total of 564 complete data sets (return rate 90.1%) from medical students and 29 questionnaires (return rate 96.7%) from general practitioners. In relation to the reference groups defined by Reis (1997), medical students had poor ambiguity tolerance on all three scales. No differences were found between those in the first and the sixth academic years, although we did observe gender-specific differences in ambiguity tolerance. We found no differences in ambiguity tolerance between general practitioners and medical students.

CONCLUSIONS

The ambiguity tolerance of the students that we assessed was below average, and appeared to be stable throughout the course of their studies. In contrast to our expectations, the general practitioners did not have a higher level of ambiguity tolerance than the students did.

Using tree diagrams without numerical values in addition to relative numbers improves students' numeracy skills: a randomized study in medical education

BACKGROUND

Physicians and medical students may lack sufficient numeracy skills to make treatment decisions, interpret test results, and practice evidence-based medicine. We evaluated whether the use of a tree diagram without numerical values as an aid for numerical processing might improve students' test results when dealing with percentages.

METHODS

A prospective randomized study was carried out with 102 third-year students. Participants received 3 diagnostic test problems and were asked to determine positive predictive values. The information in these tests was expressed either in (1) natural frequencies, (2) conditional probabilities, or (3) conditional probabilities with a tree diagram without numbers.

RESULTS

Ninety-eight (96.1%) complete data sets could be obtained. The group working with natural frequencies achieved significantly higher test results (n = 29, mean score: 1.1, P = 0.045) than the group working with conditional probabilities (n = 34, mean score: 0.56). The students who were given a tree diagram in addition to conditional probabilities (n = 35, mean score: 1.26) also achieved significantly better scores than the group with conditional probabilities alone (P = 0.008). The difference between the group who had received natural frequencies and the group working with conditional probabilities and the tree diagram was not significant.

CONCLUSIONS

We suggest the use of a tree diagram as a visual aid when dealing with diagnostic tests expressed in conditional probabilities.

Teaching ultrasound in a curricular course according to certified EFSUMB standards during undergraduate medical education: a prospective study

BACKGROUND

As a non-invasive and readily available diagnostic tool, ultrasound is one of the most important imaging techniques in medicine. Ultrasound is usually trained during residency preferable according to German Society of Ultrasound in Medicine (DEGUM) standards. Our curriculum calls for undergraduate training in ultrasound of medical students in their 4th year of undergraduate education. An explorative pilot study evaluated the acceptance of this teaching method, and compared it to other practical activities in medical education at Muenster University.

METHODS

240 medical students in their 4th year of undergraduate medical education participated in the training and completed a pre- and post-questionnaire for self-assessment of technical knowledge, self-assurance of the procedure, and motivation in performing ultrasound using a Likert scale. Moreover, students were asked about their interest in pursuing a career in internal medicine. To compare this training to other educational activities a standardized online evaluation tool was used. A direct observation of procedural skills assessment (DOPS) for the first time applied on ultrasound aimed to independently assess the success of our teaching method.

RESULTS

There was a significant increase in technical knowledge and self-assurance (p < 0.001) of the students' self-assessments. The clinical relevance and self-motivation of the teaching were evaluated positively. The students' DOPS results demonstrated proficiency in the understanding of anatomic structures shown in ultrasonographic images, including terminology, machine settings, and transducer frequencies.

CONCLUSIONS

Training ultrasound according to certified DEGUM standards was successful and should be offered in undergraduate medical education. The evaluation of the course affirmed the necessity, quality and clinical relevance of the course with a top ranking score of hands-on training courses within the educational activities of the Medical Faculty of Muenster.

[Familial occurrence of chronic tension headache]

Chronic tension-type headache occurs in 3% of the Danish population. the aetiology remains an enigma, even though it is one of the most frequent illnesses with chronic pain. A family study indicates the importance of genetic factors. Compared with the general population, first degree relatives (parents, siblings and children) of probands with chronic tension-type headache have a three-fold significantly increased risk of chronic tension-type headache, while spouses had no increased risk. Complex segregation analysis indicates multifactorial inheritance. Thus, a combination of genetic and environmental factors causes chronic tension-type headache.