Towards outcome-based programme evaluation: using student comparative self-assessments to determine teaching effectiveness

Implementing spiritual care education into the teaching of palliative medicine: an outcome evaluation

BACKGROUND

The concept of "total pain" plays an important role in palliative care; it means that pain is not solely experienced on a physical level, but also within a psychological, social and spiritual dimension. Understanding what spirituality entails, however, is a challenge for health care professionals, as is screening for the spiritual needs of patients.

OBJECTIVE

This is a novel, interprofessional approach in teaching undergraduate medical students about spiritual care in the format of a seminar. The aim of this study is to assess if an increase in knowledge about spiritual care in the clinical context is achievable with this format.

METHODS

In a mandatory seminar within the palliative care curriculum at our university, both a physician and a hospital chaplain teach strategies in symptom control from different perspectives (somatic domain - spiritual domain). For evaluation purposes of the content taught on the spiritual domain, we conducted a questionnaire consisting of two parts: specific outcome evaluation making use of the comparative self-assessment (CSA) gain and overall perception of the seminar using Likert scale.

RESULTS

In total, 52 students participated. Regarding specific outcome evaluation, the greatest gain was achieved in the ability to define total pain (84.8%) and in realizing its relevance in clinical settings (77.4%). The lowest, but still fairly high improvement was achieved in the ability to identify patients who might benefit from spiritual counselling (60.9%). The learning benefits were all significant as confirmed by confidence intervals. Overall, students were satisfied with the structure of the seminar. The content was delivered clearly and comprehensibly reaching a mean score of 4.3 on Likert scale (4 = agree). The content was perceived as overall relevant to the later work in medicine (mean 4.3). Most students do not opt for a seminar solely revolving around spiritual care (mean 2.6).

CONCLUSIONS

We conclude that implementing spiritual care education following an interprofessional approach into existing medical curricula, e.g. palliative medicine, is feasible and well perceived among medical students. Students do not wish for a seminar which solely revolves around spiritual care but prefer a close link to clinical practice and strategies.

Factors affecting medical students' satisfaction with online learning: a regression analysis of a survey

BACKGROUND

Medical education requires the implementation of different teaching methods and strategies for future doctors to achieve broad learning objectives. This wide range of methods and strategies includes the use of Information Technologies. For a long time, there was a call for a change in medical education for blending new teaching approaches to lessen medical students' class time. The COVID-19 pandemic then sped up the transition to the new way of medical education and classroom lectures were quickly moved to a virtual environment. We expect that these changes will continue, and online learning will be one of the main teaching strategies in medical education. Therefore, educational experiences during the COVID-19 pandemic will improve our understanding of online learning and will help to develop blended medical school curricula in the future. For this reason, we aimed to determine students' overall satisfaction with their online learning experience and to define the main factors affecting students' satisfaction with their online learning program at Cerrahpaşa Medical Faculty.

METHODS

A cross-sectional survey study was conducted to determine medical students' overall satisfaction with online learning methods and to identify factors associated with positive and negative satisfaction levels. A questionnaire, consisting of 24 questions to collect demographic characteristics, factors associated with online education experience and overall satisfaction levels was developed and distributed to 1600 medical students. Multivariable linear regression analysis was used to determine the factors associated with positive and negative satisfaction levels.

RESULTS

Regression analysis showed that being familiar with online teaching techniques (β = 0.19, 95% CI [0.07, 0.30], faculty members' higher online teaching skill levels (β = 0.42, 95% CI [0.32, 0.51], interactive online teaching approaches (β = 0.54, 95% CI [0.41, 0.67], having a personal workspace (β = 0.43, 95% CI [0.19, 0.67], and a self-reported longer attention span (β = 0.75, 95% CI [0.57, 0.92] were associated with higher overall satisfaction with online learning. The occurrence of technical problems (β = -0.19, 95% CI [-0.26, -0.12] was associated with lower overall satisfaction.

CONCLUSIONS

Higher online teaching skills of faculty members, use of interactive approaches, students' familiarity with online teaching techniques, provision of a personal workspace, and self-reported longer attention spans positively contributed to higher levels of student satisfaction with online learning. Considering the increasing significance of online educational methods, our study identified key components that affect students' level of satisfaction. This information might contribute to the development of online educational programs in the future.

A 1-Day Training Course to Disseminate the BREF Psychoeducational Program to Caregivers and Promote Network Establishment between Psychiatry Departments and Family Associations

OBJECTIVE

Although international guidelines state that psychoeducation to caregivers should be provided systematically, it remains insufficiently available in psychiatry. This study reports the development and evaluation of an original training course aimed to provide participants with the knowledge and skills to implement "BREF," a psychoeducational program for caregivers.

METHODS

The BREF program training course, a free, 1-day course incorporating peer role-play was developed. In addition to psychiatrists, nurses, and psychologists, caregivers were involved as preceptors. Participants were mental health professionals and volunteer caregivers. Participants to the first 28 sessions of the course (n=467) completed a post-course questionnaire (n=341) and a cross-sectional questionnaire (n=56). Quantitative data on satisfaction, learning, and behavior changes following the course were collected equating to levels 1, 2, and 3 of Kirkpatrick's model.

RESULTS

After the course, high levels of satisfaction and commitment were observed with 100% of responders recommending the course and 81% intending to implement the BREF program. Confidence mean score to implement BREF was 7.9/10 (±1.4) with no significant effect of course session. At cross-sectional evaluation, 73% of responders reported improvements in skills related to providing psychoeducation to caregivers, 64% stated that the BREF program was implemented/under implementation, and 66% stated that their department had connected with a family association.

CONCLUSIONS

Training course sessions alone can increase psychoeducational programs for caregivers and network establishment. The BREF program training course demonstrates a high level of participant satisfaction and is a promising method to disseminate psychoeducation to caregivers, thus addressing a major shortage in mental health organization.

Community-supported teaching on the topic of transgender identity in undergraduate medical education - a pilot project

Introduction and objectives

Future physicians are insufficiently prepared for the topic of transgender identity during their studies. Relevant courses during undergraduate medical education are heterogeneous and not widely established within the curricula. At our university, we investigated if students' knowledge of transgender identity could be increased through medical specialist teaching and teaching delivered by representatives of the trans* community (community-supported teaching).

Methods

During summer semester 2021 (SS21), the knowledge level on transgender identity of 134 medical students in their fifth clinical semester was evaluated (phase 1). In addition, knowledge gain on gender incongruence through the module "psychiatry" was retrospectively surveyed across two dimensions: 1. diagnostic criteria, 2. treatment/care. During winter semester 2021/22 (WS 21/22), a 90-minute seminar on transgender identity was held either community-supported or by medical specialists (phase 2). Following the psychiatry exam, a re-evaluation was carried out by 115 students (phase 3).

Results

The students in SS21 did not feel sufficiently educated in the topic of transgender identity through their studies, but rated the relevance of the topic for their later profession as high. Learning gain improved after the introduction of the seminar in WS21/22 compared to the previous semester (both dimensions p<.001). Community-supported and specialist teaching achieved equivalent results.

Conclusion

One 90-minutes seminar led to a significant learning gain regarding the topic of transgender identity. Community-supported teaching is a promising way to impart knowledge in a qualified manner: Medical faculties should use this form of teaching to convey established knowledge to students in future curricula.

Barriers and Solutions to Successful Problem-Based Learning Delivery in Developing Countries - A Literature Review

Problem-based learning (PBL) was introduced in the 1960s as an alternative to traditional teacher-centered and discipline-based preclinical medical education. A literature review was conducted to explore the barriers and solutions to successful PBL uptake and delivery in developing countries. The review involved the search of articles and scientific studies on PubMed, The Lancet, and Scielo. The review focused on the medical education literature, using as a primary search criterion "problem-based learning" in combination with "developing countries" and "education". The search was limited to articles in Spanish and English published between 2011 and November 2021, except for three articles due to their relevance to the subject. Faculty development programs are the cornerstone when implementing a new methodology in developing countries. Early career development, PBL methodology, and the available assessment options should be the primary learning objectives of these programs. Stakeholders will need to plan using available resources following the experience of other countries and institutions encouraging collaborative development. Evaluation and assessment will be crucial to understand the impact of PBL, and considerations should be taken to implement an integrated curriculum. Medical Education Research should be encouraged, appraised, and disseminated to improve evidence-based decision-making, creating a constant development cycle. PBL is innovative and represents many unanswered questions that will develop in the following decade as more schools implement new methodologies and Research on PBL.

The outcomes of postgraduate palliative care education and training: assessment and comparison of nurses and physicians

BACKGROUND

Within Germany, there is a heterogeneous range of training and continuing education in palliative care for different professional groups. The German Society for Palliative Medicine (DGP), together with the German Hospice and Palliative Care Association (DHPV), have defined quality requirements for postgraduate training in palliative care. These requirements include the evaluation of course structures and the assessment of outcomes.

AIM

To assess the 'learning gains' in palliative care nurses and physicians undertaking continuing education programmes, and evaluate the structures and processes. To identify if/how the continuing education programmes could be improved.

MATERIAL AND METHODS

The development of Nurses' and Physicians' learning was determined using a retrospective self-assessment procedure. The evaluation was based on learning objectives developed in the DGP Education Working Group, using a six-point Likert scale for each item, and space for 'free-text' comments. Assessments were conducted after training.

RESULTS

Five hundred twenty nine self-assessments were recorded (456 nurses / 73 physicians). An increase in learning is demonstrated in all areas (knowledge, skills, social and self-competence) for each profession. The greatest gain was in symptom control. However, there were significant differences in the extent of learning gains between nurses and physicians.

CONCLUSION

Analysis suggests current training results in improvements, but personal competences progress less than knowledge and skills. One way to improve this would be to introduce more interprofessional continuing education elements. Evaluation, as a basis for improving training concepts, is essential for continual development.

Caution with competitive gamification in medical education: unexpected results of a randomised cross-over study

BACKGROUND

To intrinsically motivate students in the long term, longitudinal e-learning systems combined with repeated testing and competitive gamification seem promising. The effects of this approach have never been closely examined in the field of evidence-based medicine. The authors investigated if a simple, competitive learning application enhances students' risk competence and intrinsic motivation.

METHODS

Participants were 5.-9. semester medical students (n = 48), recruited in an elective evidence-based medicine subject and randomly distributed to two groups (group 1: n = 23; group 2: n = 25). Both accessed a competitive evidence-based medicine quiz game. Following a cross-over design, each group practiced with one of two thematically different questionnaires A or B, before the allocation switched after one month. To analyse whether there was a measurable learning effect in the practiced topics, a paired t-test was performed with quantitative data from 3 e-tests. Students further reported their experience in evaluation surveys.

RESULTS

Students' improvements in e-test scores after training with the corresponding topics in the learning application can be attributed to chance. Even though the majority enjoyed playing and felt motivated to study, they invested a minimum of time and rejected competition.

CONCLUSION

The authors found no evidence for benefits of the investigated learning programme on students' risk competence or on their internal motivation. The majority disapproved the competitive concept, indicating adverse side effects of the applied gamification element. To intrinsically motivate more students, prospective learning programmes should favour complex and collaborative programmes over simple and competitive ones.

Mindfulness and compassion training on daily work with patients and within the multiprofessional palliative care team: a retrospective self-assessment study

BACKGROUND

Palliative care teams work under challenging conditions in a sensitive setting with difficult tasks. The multi-professional team can play an important role. Mindfulness and compassion-based practices are used to build resilience. Our aim was to examine (1) feasibility and acceptability, (2) satisfaction and impact, and (3) opportunities and limitations of a mindfulness course.

METHODS

An eight-week mindfulness and compassion course was delivered in a university-based specialized palliative care unit. A meditation teacher provided preparatory evening sessions and meditation exercises that could be integrated into daily activities. The scientific analysis of the course was based on a questionnaire developed for quality assessmentThe first two parts consisted of demographic, Likert-type, and free-text items. Part 3 consisted of learning objectives that were self-assessed after finishing the course (post-then). In the analysis, we used descriptive statistics, qualitative content analysis, and comparative self-assessment.

RESULTS

Twenty four employees participated. 58% of participants attended 4 or more of the 7 voluntary mindfulness days. 91% expressed moderate to high satisfaction and would recommend the palliative care program to others. Three main categories emerged in the qualitative content analysis: providing feedback on the course, personal impact, and impact on professional life. The opportunity for self-care in a professional context was highlighted. Learning gains (CSA Gain) were high (38.5-49.4%) in terms of knowledge and techniques, moderate (26.2-34.5%) in terms of implementation of learned skills, and rather low (12.7-24.6%) in terms of changes to attitude.

CONCLUSION

Our evaluation shows that the participants of a mindfulness and compassion course considered it as a feasible and welcome tool to familiarize a multi-professional palliative care team with self-care techniques.

TRIAL REGISTRATION

Internal Clinical Trial Register of the Medical Faculty, Heinrich Heine University Düsseldorf, No. 2018074763 (registered retrospectively on 30^th July 2018).

Bridging vascular physiology to vascular medicine: an integrative laboratory class

Vascular diseases of the legs are highly prevalent and constitute an important part of medical curricula. The understanding of these diseases relies on strongly interwoven aspects of vascular physiology and vascular medicine. We aimed to connect these within a horizontally integrated laboratory class on vascular physiology of the leg that was designed in cooperation between the departments of physiology and vascular surgery. Conceptually, we applied examination techniques of vascular medicine to visualize physiological parameters that are altered by the most frequent diseases. This facilitates integrative discussions on malfunctions, trains diagnostic skills, and bridges to vascular medicine. In four experiments, we use oscillometry and impedance venous occlusion plethysmography to address key aspects of the arterial and venous system of the legs: 1) arterial pulse wave, 2) arterial systolic blood pressure, 3) venous capacitance and venous outflow, and 4) reactive hyperemia. After the experiments, physiological vascular function, the associated diseases, their impact on the recorded parameters, and diagnostic options are discussed. To allow reproduction, we describe the course structure and the experimental setup in detail. We present the experimental data of a cohort of medical students and document learning success and student satisfaction. All experiments were feasible and provided robust data on physiologically and clinically relevant vascular functions. The activity was perceived positively by the students and led to a substantial improvement of knowledge. With this work, we offer a template for reproduction or variation of a proven concept of horizontally integrated teaching of vascular physiology of the leg.NEW & NOTEWORTHY This article presents an integrative laboratory class on vascular physiology bridging to vascular medicine. The four experiments rely on oscillometry and venous occlusion plethysmography. We describe in detail this new class regarding structure, experimental setup, and experimental procedure, and we give insight into the applied materials. Moreover, we present the experimental data of 74 students and a quantitative evaluation of the students' learning success and acceptance.

[Portfolio-based learning in surgery : Making competencies "visible"]

BACKGROUND

Due to an increasing competence orientation of medical studies, surgical curricula are being adapted in many places. In addition to surgical knowledge and practical skills, these should also teach competencies in differential diagnostics and treatment. The teaching of surgical knowledge through lectures and seminars and the demonstration of practical skills, e.g., through the use of logbooks in the Bock Practical Surgery (BP), only allows limited active engagement with surgical competencies on differential diagnostics and treatment. A reflection-based portfolio allows, through the independent written elaboration of surgical topics, an active engagement with the competencies and promises a higher learning effect. In the context of the implementation of such a portfolio as part of the proof of activity in BP, the effects on the acquisition of competencies and on the way of learning were investigated.

MATERIAL AND METHODS

Using a mixed methods approach, we compared competence acquisition using a reflection-based portfolio with learning using a logbook. Students conducted a self-assessment of competencies using questionnaire surveys before and after the BP. Through focus group interviews with discussions among students using a guideline, we explored the different ways of acquiring competencies. In addition, the examination and evaluation results of both cohorts were compared.

RESULTS AND DISCUSSION

Students' self-assessed competency acquisition and examination and evaluation results showed no differences when comparing the two cohorts. During the focus group interviews, we were able to show that in the perception of the students, surgical competencies can be made more visible and thus more explicit with the help of a reflection-based portfolio. In addition, self-regulated learning was promoted without neglecting practical skills. Students demanded greater supervision and guidance by mentors in both groups.

Integrated clinical case discussions - a fully student-organized peer-teaching program on internal medicine

BACKGROUND

In response to students´ poor ratings of emergency remote lectures in internal medicine, a team of undergraduate medical students initiated a series of voluntary peer-moderated clinical case discussions. This study aims to describe the student-led effort to develop peer-moderated clinical case discussions focused on training cognitive clinical skill for first and second-year clinical students.

METHODS

Following the Kern Cycle a didactic concept is conceived by matching cognitive learning theory to the competence levels of the German Medical Training Framework. A 50-item survey is developed based on previous evaluation tools and administered after each tutorial. Educational environment, cognitive congruence, and learning outcomes are assessed using pre-post-self-reports in a single-institution study.

RESULTS

Over the course of two semesters 19 tutors conducted 48 tutorials. There were 794 attendances in total (273 in the first semester and 521 in the second). The response rate was 32%. The didactic concept proved successful in attaining all learning objectives. Students rated the educational environment, cognitive congruence, and tutorials overall as "very good" and significantly better than the corresponding lecture. Students reported a 70%-increase in positive feelings about being tutored by peers after the session.

CONCLUSION

Peer-assisted learning can improve students´ subjective satisfaction levels and successfully foster clinical reasoning skills. This highlights successful student contributions to the development of curricula.

Palliative Care e-Learning for Physicians Caring for Critically Ill and Dying Patients during the COVID-19 Pandemic: An Outcome Evaluation with Self-Assessed Knowledge and Attitude

During the COVID-19 pandemic, the care of critically ill and dying patients in isolation wards, intensive care units (ICUs), and regular wards was severely impaired. In order to support physicians in communicative and palliative care skills, an e-learning tool was developed as part of the joint project "Palliative Care in Pandemic Times" (PallPan). This study investigates the feasibility of this e-learning tool. Secondly, we aim to analyze changes in knowledge and attitude upon completion of the e-learning tool. A 38-item questionnaire-based evaluation study with assessment of global and specific outcomes including ICU and non-ICU physicians was performed. In total, 24 questionnaires were included in the anonymous analysis. Feasibility was confirmed by a very high rate of overall satisfaction (94% approval), with relevance reaching 99% approval. Overall, we detected high gains in knowledge and noticeably lower gains on the attitude plane, with the highest gain in naming reasons for incorporating palliative care. The lowest learning gain on the attitude plane was observed when the participants were confronted with their own mortality. This study shows that e-learning is a feasible tool for gaining knowledge and even changing the attitudes of physicians caring for critically ill and dying patients in a self-assessment evaluation.

Virtual reality as a teaching method for resuscitation training in undergraduate first year medical students during COVID-19 pandemic: a randomised controlled trial

BACKGROUND

Virtual reality (VR) is a computer-generated simulation technique which yields plenty of benefits and its application in medical education is growing. This study explored the effectiveness of a VR Basic Life Support (BLS) training compared to a web-based training during the COVID-19 pandemic, in which face-to-face trainings were disrupted or reduced.

METHODS

This randomised, double-blinded, controlled study, enrolled 1^st year medical students. The control group took part in web-based BLS training, the intervention group received an additional individual VR BLS training. The primary endpoint was the no-flow time-an indicator for the quality of BLS-, assessed during a structural clinical examination, in which also the overall quality of BLS (secondary outcome) was rated. The tertiary outcome was the learning gain of the undergraduates, assessed with a comparative self-assessment (CSA).

RESULTS

Data from 88 undergraduates (n = 46 intervention- and n = 42 control group) were analysed. The intervention group had a significant lower no-flow time (p = .009) with a difference between the two groups of 28% (95%-CI [8%;43%]). The overall BLS performance of the intervention group was also significantly better than the control group with a mean difference of 15.44 points (95%-CI [21.049.83]), p < .001. In the CSA the undergraduates of the intervention group reported a significant higher learning gain.

CONCLUSION

VR proved to be effective in enhancing process quality of BLS, therefore, the integration of VR into resuscitation trainings should be considered. Further research needs to explore which combination of instructional designs leads to deliberate practice and mastery learning of BLS.

Blended teaching of medical ethics during COVID-19: practice and reflection

BACKGROUND

With the advancement of information technology, teachers have become able to overcome the limitations of time and room capacity by carrying out teaching activities online. This practice, however, also presents new challenges. The present study explores how to fully capitalize on the advantages of online and offline teaching and improve the quality and impact of the teaching delivered. This article presents an analysis of the planning, implementation, evaluation, and reflection process of reforming the Fujian Medical University (FJMU) medical ethics course.

METHODS

After early attempts using the Small Private Online Course (SPOC) and flipped classroom formats, this paper focuses on the comprehensive active implementation of blended teaching practice. In terms of teaching practice, this research makes targeted improvements to overcome the known shortcomings of SPOCs and flipped classrooms, including the significant preparatory workload and lacking enthusiasm for classroom participation, by redesigning the course and evaluation method and changing the role of the teacher in blended teaching. Subsequently, the study used a stratified sampling method to select 20 students enrolled in the clinical medicine course at Fujian Medical University (FJMU). Their course experience was investigated using a semi-structured interview. Interview content related to evaluating teaching effect was extracted and encoded for subsequent qualitative analysis.

RESULTS

A qualitative analysis of the student evaluation of blended teaching as implemented on the medical ethics course showed that the main factors influencing student engagement are the method of assigning tasks and that of testing learning outcomes. Student participation in class is influenced by the richness of the curriculum resources available and the role played by the teacher.

CONCLUSION

This research presents a discussion of blended teaching and suggests improvements that can be made to address low student engagement and poor classroom participation. This round of blended teaching was shown to improve learning autonomy and classroom participation and to support students in the development of their clinical abilities and higher-order thinking skills. These findings provide a reference for the implementation of online teaching during the COVID-19 pandemic.

[Project report on fostering scientific competencies in pain medicine in the context of student education]

AIM

The training of scientific skills and competencies is an essential part of academic medical studies. As part of the MaReCuM model study program at Heidelberg University's Mannheim Medical School, a fifth-year rotation on scientific skills in the field of pain medicine was implemented. This paper describes this competence-oriented rotation as well as the investigation of the educational effect.

METHOD

A total of 114 fifth-year medical students participated in the survey (response rate: 83%). The control group completed the fifth year prior to the implementation of the rotation. The experimental group was required to participate in the rotation and the real healthcare research study "Case management program: low back pain". A survey of both groups was conducted on the first day of the rotation and at the end of the module.

RESULTS

The innovative and competency-based learning unit was successfully implemented as part of the MaReCuM model study program and carried out with partners in general practice as well as the Mannheim Institute of Public Health. The participating students accepted the rotation well. There was no measurable effect on the subjective learning success of the rotation in the evaluation.

DISCUSSION

To the authors' knowledge, this educational approach has never been tested before in a German study program. The presented rotation offers an additional option for the training of scientific competencies as part of medical studies. The missing of a measurable effect could be due to the extensive experience of the medical students as well as the limitations on participation in a real healthcare study. An additional learning opportunity could be created by connecting the preexisting lectures to a longitudinal module on scholarly competencies. The implementation of the program also offers a unique opportunity for educational research on the acquisition of scientific competencies in medical students.

Communication in the face of death and dying - how does the encounter with death influence the patient management competence of medical students? An outcome-evaluation

BACKGROUND INFORMATION

As part of an elective course, the Interdisciplinary Centre for Palliative Medicine at Duesseldorf University Hospital offers medical students the opportunity to personally meet and talk to a seriously ill patient on one or more occasions. The future physicians are provided with an opportunity to broaden their professional competence, i.e. their knowledge and skills in patient-centred communication at the end of life, and enhance their personal competence, for example in how to professionally handle their own emotions. A topical e-learning module helps the students to prepare for the meetings, and writing a reflection paper forms the basis for the concluding reflection seminar.

OBJECTIVES

The study's objective is a global and outcome-based evaluation of the elective blended-learning course that provides real-world patient interaction. The outcome-based evaluation or outcome assessment aims to objectively evaluate changes identified in knowledge, skills and attitude among the participants of the elective-course. Furthermore, the evaluation aims to answer the question of whether changes especially in attitude (social skills and self-competence) should be expected after the students have met with severely ill or dying patients.

METHOD

On two questionnaires specifically developed for this survey the students were able to provide a global rating of the elective course and describe their learning gains in palliative care. The students' learning gains were measured by means of 14 items reflecting the specific educational objectives of the offered elective course. Using the German school grading system as a rating scale, the students assessed their learning progress by retrospectively evaluating their skills before and after completion of the elective course (Comparative Self-Assessment, CSA).

RESULTS

In the time from April 2018 till March 2020, 62 students participated in the evaluation. Overall, learning progress among students could be observed across all areas of competence, and in 50% of all retrospective self-assessment items the learning gains were ≥ 50%. The highest learning gain (63.6%) was observed in the students' ability to meet a severely ill patient without fear. The lowest learning gain was observed when students had to confront and accept their own mortality.

CONCLUSIONS

The offered elective course supports students in achieving social and self-competence development goals. According to the obtained results, contact with real-world patients helps mould the students' attitude.

A Weighted Evaluation Study of Clinical Teacher Performance at Five Hospitals in the UK

INTRODUCTION

Evaluation of individual teachers in undergraduate medical education helps clinical teaching fellows identify their own strengths and weaknesses. In addition, evaluation data can be used to guide career decisions. In order for evaluation results to adequately reflect true teaching performance, a range of parameters should be considered when designing data collection tools.

METHODS

Clinical teaching fellows at five London teaching hospitals were evaluated by third-year students they had supervised during a ten-week clinical attachment. The questionnaire addressed (a) general teaching skills and (b) student learning outcome measured via comparative self-assessments. Teachers were ranked using different algorithms with various weights assigned to these two factors.

RESULTS

A total of 133 students evaluated 14 teaching fellows. Overall, ratings on teaching skills were largely favourable, while the perceived increase in student performance was modest. Considerable variability across teachers was observed for both factors. Teacher rankings were strongly influenced by the weighting algorithm used. Depending on the algorithm, one teacher was assigned any rank between #2 and #10.

CONCLUSION

Both parts of the questionnaire address different outcomes and thus highlight specific strengths and weaknesses of individual teachers. Programme directors need to carefully consider the weight assigned to individual components of teacher evaluations in order to ensure a fair appraisal of teacher performance.

A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

BACKGROUND

Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.

METHODS

Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.

RESULTS

Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values.

CONCLUSIONS

Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

Virtual reality as a teaching method for resuscitation training in undergraduate first year medical students: a randomized controlled trial

Background

Virtual reality is an innovative technology for medical education associated with high empirical realism.

Therefore, this study compares a conventional cardiopulmonary resuscitation (CPR) training with a Virtual Reality (VR) training aiming to demonstrate: (a) non-inferiority of the VR intervention in respect of no flow time and (b) superiority in respect of subjective learning gain.

Methods

In this controlled randomized study first year, undergraduate students were allocated in the intervention group and the control group. Fifty-six participants were randomized to the intervention group and 104 participants to the control group. The intervention group received an individual 35-min VR Basic Life Support (BLS) course and a basic skill training. The control group took part in a “classic” BLS-course with a seminar and a basic skill training.

The groups were compared in respect of no flow time in a final 3-min BLS examination (primary outcome) and their learning gain (secondary outcome) assessed with a comparative self-assessment (CSA) using a questionnaire at the beginning and the end of the course. Data analysis was performed with a general linear fixed effects model.

Results

The no flow time was significantly shorter in the control group (Mean values: control group 82 s vs. intervention group 93 s; p = 0.000). In the CSA participants of the intervention group had a higher learning gain in 6 out of 11 items of the questionnaire (p < 0.05).

Conclusion

A “classic” BLS-course with a seminar and training seems superior to VR in teaching technical skills. However, overall learning gain was higher with VR. Future BLS course-formats should consider the integration of VR technique into the classic CPR training or vice versa, to use the advantage of both teaching techniques.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13049-021-00836-y.

Restructuring the clinical curriculum at University Medical Center Göttingen: effects of distance teaching on students' satisfaction and learning outcome

Introduction: In summer term 2020, the clinical phase of the undergraduate medical curriculum at University Medical Center Göttingen was restructured since distance teaching had to be used predominantly due to contact restrictions during the COVID-19 pandemic. This paper investigates the impact of restructuring the clinical curriculum on medical students' satisfaction and learning outcomes. Methods: In each cohort, the 13-week curriculum was divided into two parts: During the first 9 weeks, factual knowledge was imparted using distance teaching by means of a modified inverted classroom approach. This was followed by a 4-week period of adapted classroom teaching involving both real and virtual patients in order to train students' practical skills. The evaluation of the 21 clinical modules comprised students' satisfaction with distance teaching as well as students' learning outcome. The latter was assessed by means of comparative self-assessment (CSA) gain and the results of the module exams, respectively. Data of summer term 2020 (= distance teaching, DT) were compared with respective data of winter term 2019/20 (= classroom teaching, CT) and analysed for differences and correlations. Results: Response rates of evaluations were 51.3% in CT and 19.3% in DT. There was no significant difference between mean scores in module exams in CT and DT, respectively. However, CSA gain was significantly lower in DT (p=0.047) compared with CT. Further analyses revealed that CSA gain depended on the time point of data collection: CSA gain was lower the more time had passed since the end of a specific module. Moreover, we found positive correlations between CSA gain and students' satisfaction with various aspects of distance teaching, particularly with "communication between teachers and students" (rho=0.674; p=0.002). Discussion and conclusions: Although some limitations and confounding factors have to be taken into account (such as evaluation response rates, assessment time points, and proportion of familiar items in module exams), the following recommendations can be derived from our findings: A valid assessment of students' learning outcome by means of exam results requires that as few exam items as possible are familiar to the students. CSA gain seems to be valid if assessment time points are standardised and not contaminated by students' learning activities for other modules. Good communication between teachers and students may contribute to increase students' satisfaction with distance teaching.

Presentation and evaluation of the teaching concept "ENHANCE" for basic sciences in medical education

A solid understanding of basic sciences is a prerequisite for successful completion of medical education. Therefore, it is essential to improve the quality of teaching and to ensure the applicability of basic sciences. Based on practical experiences and previous research, we developed an innovative step-by-step concept, called ENHANCE, for the implementation or revision of teaching units, especially for basic sciences. We used comparative self-assessment gains, a questionnaire to assess teaching quality as well as end-of-semester evaluations (students' satisfaction and open-ended questions) to evaluate the ENHANCE concept. It was found that ENHANCE-based teaching units were related to increased students' satisfaction, high attendance rates and that restructuring the course curriculum yielded in a positive assessment of teaching effectiveness. The revised courses were rated as the very best of all classes in several semesters. Qualitative data showed that students particularly appreciated the level of comprehension and how helpful the courses were for the understanding and preparation of the regular curriculum.

One-week multidisciplinary post-graduate palliative care training: an outcome-based program evaluation

BACKGROUND

A multi-professional, post-graduate, one-week palliative care training program was piloted in November 2019 at the University of Ivano-Frankivsk, Ukraine. A formal evaluation of this program was performed.

METHODS

This is a comparative, retrospective outcome-based evaluation of an educational intervention. Participants completed evaluation forms at the end of the course (post-intervention = T1), covering demographics, comparative retrospective self-assessment (40 items, 6-point Likert scale), organizational aspects, and general feedback (free text). At T1, the responses represent actual self-assessment, pre-interventional (T0) scores were generated by retrospective self-assessment. The Retrospective Performance Gain (RPG) was calculated on group level for the comparative self-assessment, demographic and organizational variables were analyzed by descriptive statistics, and free text answers were processed by qualitative methodology (content analysis).

RESULTS

Fifty-three of 56 attendants from all professions relevant to palliative care completed the evaluation forms (response 94,6%), with mean age 39y (22-64) and mean working experience 13,6y (1-44). Overall ratings of the program were very positive. Comparative retrospective self-assessment demonstrated a marked RPG from T0 to T1 on all items. Free text comments emphasized the need for regular nation-wide educational programs and for further education in bereavement care; inter-professional practice; communication; palliative care philosophy; professional self-care; specific nursing skills; dementia care; and advocacy, while the general contribution of the program to palliative care development in Ukraine was acknowledged.

CONCLUSIONS

Systematic evaluation of a post-graduate international training program in palliative care may provide a mutual learning experience and map country-specific barriers and facilitators that have to be addressed when setting up palliative care services.

Influence of the wording of evaluation items on outcome-based evaluation results for large-group teaching in anatomy, biochemistry and legal medicine

Student learning outcome is an important dimension of teaching quality in undergraduate medical education. Measuring an increase in knowledge during teaching requires repetitive objective testing which is usually not feasible. As an alternative, student learning outcome can be calculated from student self-ratings. Comparative self-assessment (CSA) gain reflects the performance difference before and after teaching, adjusted for initial knowledge. It has been shown to be a valid proxy measure of actual learning outcome derived from objective tests. However, student self-ratings are prone to a number of confounding factors. In the context of outcome-based evaluation, the wording of self-rating items is crucial to the validity of evaluation results. This randomized trial assessed whether including qualifiers in these statements impacts on student ratings and CSA gain. First-year medical students self-rated their initial (then-test) and final (post-test) knowledge for lectures in anatomy, biochemistry and legal medicine, respectively, and 659 questionnaires were retrieved. Six-point scales were used for self-ratings with 1 being the most positive option. Qualifier use did not affect then-test ratings but was associated with slightly less favorable post-test ratings. Consecutively, mean CSA gain was smaller for items containing qualifiers than for items lacking qualifiers (50.6±15.0% vs. 56.3±14.6%, p=0.079). The effect was more pronounced (Cohen's d=0.82) for items related to anatomy. In order to increase fairness of outcome-based evaluation and increase the comparability of CSA gain data across subjects, medical educators should agree on a consistent approach (qualifiers for all items or no qualifiers at all) when drafting self-rating statements for outcome-based evaluation.

Post mortem CT scans as a supplementary teaching method in gross anatomy

Despite increasing the integration of radiologic imaging teaching in anatomy dissection courses, studies on learning outcome of these interventions are rare or have certain shortcomings in study design. In this study, students were randomly allocated to an intervention group (n=53) receiving five weekly CT-courses of 30min duration during a 6-week gross anatomy course. Students in the control group (n=329) received no additional teaching. Total teaching time did not differ among groups. All students were asked to participate in a pre- and post-course self-assessment (comparative self-assessment; CSA) of learning objectives related to anatomical spatial relationships and a post-course formative assessment on radiologic anatomy. Items of both assessments were matched. Moreover, students of the intervention group were asked to evaluate the CT-courses. Most participants of the intervention group classified the CT-courses as "good" or "very good". Nevertheless, results of the CSA and formative assessment did not differ among study and control groups. These findings indicate that the teaching intervention (CT-courses) did not have an impact on recognition of anatomical structures in radiological images beyond the knowledge acquired in the anatomical dissection course. As a consequence, interventions integrating radiology imaging into dissection courses should be based on psychological considerations of how to best foster student learning. Learning outcome has to be monitored, as results of evaluation surveys can be misleading. Further research on curricular concepts is needed considering both short- and long-term effects.

A pilot Tuning Project-based national study on recently graduated medical students' self-assessment of competences--the TEST study

BACKGROUND

The Tuning Project is an initiative funded by the European Commission that developed core competences for primary medical degrees in Europe. Students' grouped self-assessments are used for program evaluation and improvement of curricula. The TEST study aimed to assess how do Portuguese medical graduates self-assess their acquisition of core competences and experiences of contact with patients in core settings according to the Tuning framework.

METHODS

Translation of the Tuning's competences (Clinical Practice - CP), Knowledge (K) items and Clinical Settings (CS) was performed. Questionnaires were created in paper and electronic formats and distributed to 1591 graduates from seven Portuguese medical schools (July 2014). Items were rated in a 6-point Likert scale (0-5) of levels of competence. Exploratory factor analysis (EFA) was conducted and Cronbach's alpha was used to evaluate the internal consistency of the questionnaire. Kruskal-Wallis and Dunn's tests were used for multiple comparisons.

RESULTS

Three hundred eighty seven questionnaires were analyzed, corresponding to 24% of the target population. EFA yielded an 11-factor solution for CP and a 6-factor solution for K items. The median value of CP factors was 2.8 (p25 = 2.0; p75 = 3.5) and the median value of K factors was 2.6 (2.0; 3.2). Factor scores ranged from 1.3 (Legal principles) to 4.0 (Ethical principles). Clinical presentations, psychological aspects of illness, evidence-based medicine and promotion of health showed the highest results. Lower scores were detected in medical emergencies, practical procedures, prescribing drugs and legal principles. More than 90% of graduates experienced having contact with patients in 8 CS but only 24% of graduates had contact in all 14 CS. Graduates had the least contact with patients in the emergency rooms, intensive care units, palliative, rehabilitation and anesthetic care. Significant differences (p < 0.05) among schools were detected in 8 factors and 7 settings.

CONCLUSIONS

We developed a valid questionnaire supporting national SWOT analysis on the acquisition of core competences in medical education. Results suggest that Portuguese graduates are not fully prepared for clinical practice. Curricular improvements in core competences and the educational development of the transition period between undergraduate and postgraduate education ought to be considered. Outcome-based program evaluation relying on graduates' grouped self-assessments contributes to inform changes in medical education.

Evaluation of competence-based teaching in higher education: From theory to practice

Competence-based teaching in higher education institutions and its evaluation have become a prevalent topic especially in the European Union. However, evaluation instruments are often limited, for example to single student competencies or specific elements of the teaching process. The present paper provides a more comprehensive evaluation concept that contributes to sustainable improvement of competence-based teaching in higher education institutions. The evaluation concept considers competence research developments as well as the participatory evaluation approach. The evaluation concept consists of three stages. The first stage evaluates whether the competencies students are supposed to acquire within the curriculum (ideal situation) are well defined. The second stage evaluates the teaching process and the competencies students have actually acquired (real situation). The third stage evaluates concrete aspects of the teaching process. Additionally, an implementation strategy is introduced to support the transfer from the theoretical evaluation concept to practice. The evaluation concept and its implementation strategy are designed for internal evaluations in higher education and primarily address higher education institutions that have already developed and conducted a competence-based curriculum.

Evaluation in medical education: A topical review of target parameters, data collection tools and confounding factors

Background and objective: Evaluation is an integral part of education in German medical schools. According to the quality standards set by the German Society for Evaluation, evaluation tools must provide an accurate and fair appraisal of teaching quality. Thus, data collection tools must be highly reliable and valid. This review summarises the current literature on evaluation of medical education with regard to the possible dimensions of teaching quality, the psychometric properties of survey instruments and potential confounding factors.

Methods: We searched Pubmed, PsycINFO and PSYNDEX for literature on evaluation in medical education and included studies published up until June 30, 2011 as well as articles identified in the “grey literature”. Results are presented as a narrative review.

Results: We identified four dimensions of teaching quality: structure, process, teacher characteristics, and outcome. Student ratings are predominantly used to address the first three dimensions, and a number of reliable tools are available for this purpose. However, potential confounders of student ratings pose a threat to the validity of these instruments. Outcome is usually operationalised in terms of student performance on examinations, but methodological problems may limit the usability of these data for evaluation purposes. In addition, not all examinations at German medical schools meet current quality standards.

Conclusion: The choice of tools for evaluating medical education should be guided by the dimension that is targeted by the evaluation. Likewise, evaluation results can only be interpreted within the context of the construct addressed by the data collection tool that was used as well as its specific confounding factors.

Assessment of two different types of bias affecting the results of outcome-based evaluation in undergraduate medical education

BACKGROUND

Estimating learning outcome from comparative student self-ratings is a reliable and valid method to identify specific strengths and shortcomings in undergraduate medical curricula. However, requiring students to complete two evaluation forms (i.e. one before and one after teaching) might adversely affect response rates. Alternatively, students could be asked to rate their initial performance level retrospectively. This approach might threaten the validity of results due to response shift or effort justification bias.

METHODS

Two consecutive cohorts of medical students enrolled in a six-week cardio-respiratory module were enrolled in this study. In both cohorts, performance gain was estimated for 33 specific learning objectives. In the first cohort, outcomes calculated from ratings provided before (pretest) and after (posttest) teaching were compared to outcomes derived from comparative self-ratings collected after teaching only (thentest and posttest). In the second cohort, only thentests and posttests were used to calculate outcomes, but data collection tools differed with regard to item presentation. In one group, thentest and posttest ratings were obtained sequentially on separate forms while in the other, both ratings were obtained simultaneously for each learning objective.

RESULTS

Using thentest ratings to calculate performance gain produced slightly higher values than using true pretest ratings. Direct comparison of then- and posttest ratings also yielded slightly higher performance gain than sequential ratings, but this effect was negligibly small.

CONCLUSIONS

Given the small effect sizes, using thentests appears to be equivalent to using true pretest ratings. Item presentation in the posttest does not significantly impact on results.

Estimating learning outcomes from pre- and posttest student self-assessments: a longitudinal study

PURPOSE

Learning outcome is an important measure for overall teaching quality and should be addressed by comprehensive evaluation tools. The authors evaluated the validity of a novel evaluation tool based on student self-assessments, which may help identify specific strengths and weaknesses of a particular course.

METHOD

In 2011, the authors asked 145 fourth-year students at Göttingen Medical School to self-assess their knowledge on 33 specific learning objectives in a pretest and posttest as part of a cardiorespiratory module. The authors compared performance gain calculated from self-assessments with performance gain derived from formative examinations that were closely matched to these 33 learning objectives.

RESULTS

Eighty-three students (57.2%) completed the assessment. There was good agreement between performance gain derived from subjective data and performance gain derived from objective examinations (Pearson r=0.78; P<.0001) on the group level. The association between the two measures was much weaker when data were analyzed on the individual level. Further analysis determined a quality cutoff for performance gain derived from aggregated student self-assessments. When using this cutoff, the evaluation tool was highly sensitive in identifying specific learning objectives with favorable or suboptimal objective performance gains.

CONCLUSIONS

The tool is easy to implement, takes initial performance levels into account, and does not require extensive pre-post testing. By providing valid estimates of actual performance gain obtained during a teaching module, it may assist medical teachers in identifying strengths and weaknesses of a particular course on the level of specific learning objectives.

German medical students' beliefs about how best to treat alcohol use disorder

BACKGROUND/AIMS

A minority of German medical students believe they know how to support smokers willing to quit. This paper examined whether the same would be true for treating alcohol use disorder (AUD), and individual factors associated with incorrect beliefs about the effectiveness of methods to treat AUD.

METHODS

In this cross-sectional study, 19,526 undergraduate students from 27 German medical schools completed a survey addressing beliefs about the effectiveness of different methods of overcoming AUD. Beliefs about AUD treatment effectiveness were compared across the 5 years of undergraduate education and predictors identified by means of multiple linear regression.

RESULTS

Even in the fifth year, 28.1% (95% CI: 26.5-29.7) of students believed that willpower alone was more effective for overcoming AUD than a comprehensive treatment program. The only significant predictor of this belief was a similar belief for stopping smoking.

CONCLUSION

Our results indicate that a considerable proportion of German medical students overestimate the effectiveness of willpower to treat smoking and AUD. The addictive nature of these disorders needs to be stressed during undergraduate medical education to ensure that future physicians will be able and motivated to support patients in their quit attempts.

Student perceptions of evaluation in undergraduate medical education: A qualitative study from one medical school

BACKGROUND

Evaluation is an integral part of medical education. Despite a wide use of various evaluation tools, little is known about student perceptions regarding the purpose and desired consequences of evaluation. Such knowledge is important to facilitate interpretation of evaluation results. The aims of this study were to elicit student views on the purpose of evaluation, indicators of teaching quality, evaluation tools and possible consequences drawn from evaluation data.

METHODS

This qualitative study involved 17 undergraduate medical students in Years 3 and 4 participating in 3 focus group interviews. Content analysis was conducted by two different researchers.

RESULTS

Evaluation was viewed as a means to facilitate improvements within medical education. Teaching quality was believed to be dependent on content, process, teacher and student characteristics as well as learning outcome, with an emphasis on the latter. Students preferred online evaluations over paper-and-pencil forms and suggested circulating results among all faculty and students. Students strongly favoured the allocation of rewards and incentives for good teaching to individual teachers.

CONCLUSIONS

In addition to assessing structural aspects of teaching, evaluation tools need to adequately address learning outcome. The use of reliable and valid evaluation methods is a prerequisite for resource allocation to individual teachers based on evaluation results.

Piloting an outcome-based programme evaluation tool in undergraduate medical education

Aims: Different approaches to performance-oriented allocation of resources according to teaching quality are currently being discussed within German medical schools. The implementation of these programmes is impeded by a lack of valid criteria to measure teaching quality. An assessment of teaching quality should include structural and procedural aspects but focus on learning outcome itself. The aim of this study was to implement a novel, outcome-based evaluation tool within the clinical phase of a medical curriculum and address differences between the novel tool and traditional evaluation methods.

Methods: Student self-assessments before and after completion of a teaching module were used to compute performance gains for specific learning objectives. Mean performance gains in each module were compared to student expectations before the module and data derived from a traditional evaluation tool using overall course ratings at the end of the module.

Results: A ranking of the 21 modules according to computed performance gains yielded entirely different results than module rankings based on overall course ratings. There was no significant correlation between performance gain and overall ratings. However, the latter were significantly correlated to student expectations before entering the module as well as structural and procedural parameters (Pearson’s r 0.7-0.9).

Conclusion: Performance gain computed from comparative self-assessments adds an important new dimension to course evaluation in medical education. In contrast to overall course ratings, the novel tool is less heavily confounded by construct-irrelevant factors. Thus, it appears to be more appropriate than overall course ratings in determining teaching quality and developing algorithms to guide performance-oriented resource allocation in medical education.

[Imatinib--a possible therapeutic option for cervical carcinoma: results of a preclinical phase I study]

BACKGROUND

In the last few years, the therapy of cervical carcinoma has progressed substantially due to the use of simultaneous platinum- containing radiochemotherapy. However, there are no data which evaluate an individualized treatment adapted to tumor biology, in spite of the fact that patients show remarkably different responses to chemotherapy. Therefore this preclinical phase I study aims at finding therapeutic alternatives to the current cytostatic drugs to treat cervical carcinoma.

MATERIAL AND METHODS

In a tumor chemosensitivity assay, 8 drugs were tested on freshly isolated tumor cells of 16 patients [carbo- and cisplatin, topotecan, paclitaxel as well as the 2 tyrosine kinase inhibitors imatinib (Glivec) and gefitinib (Iressa (R) ) and the 2 monoclonal antibodies cetuximab (Erbitux) and trastuzumab (Herceptin (R) )].

RESULTS

Overall the test was evaluable for 16 specimens (100%). Ten of 15 tumor samples (66.6%) were sensitive to imatinib. A sensitive therapeutic response could be demonstrated in all tested FIGO stages. An interindividual comparison could establish sensitivity to cetuximab in 12.5% of cases, to gefitinib in 6.25%, to trastuzumab in 6.6%, to cisplatin in 13.3%, to carboplatin in 7.6%, to paclitaxel in 93.8% and to topotecan in 25%.

CONCLUSION

Imatinib seems to be an efficacious therapeutic option for patients with cervical carcinoma, independently of tumor subtype.