A systematic review of factors influencing student ratings in undergraduate medical education course evaluations

BACKGROUND

Student ratings are a popular source of course evaluations in undergraduate medical education. Data on the reliability and validity of such ratings have mostly been derived from studies unrelated to medical education. Since medical education differs considerably from other higher education settings, an analysis of factors influencing overall student ratings with a specific focus on medical education was needed.

METHODS

For the purpose of this systematic review, online databases (PubMed, PsycInfo and Web of Science) were searched up to August 1st, 2013. Original research articles on the use of student ratings in course evaluations in undergraduate medical education were eligible for inclusion. Included studies considered the format of evaluation tools and assessed the association of independent and dependent (i.e., overall course ratings) variables. Inclusion and exclusion criteria were checked by two independent reviewers, and results were synthesised in a narrative review.

RESULTS

Twenty-five studies met the inclusion criteria. Qualitative research (2 studies) indicated that overall course ratings are mainly influenced by student satisfaction with teaching and exam difficulty rather than objective determinants of high quality teaching. Quantitative research (23 studies) yielded various influencing factors related to four categories: student characteristics, exposure to teaching, satisfaction with examinations and the evaluation process itself. Female gender, greater initial interest in course content, higher exam scores and higher satisfaction with exams were associated with more positive overall course ratings.

CONCLUSIONS

Due to the heterogeneity and methodological limitations of included studies, results must be interpreted with caution. Medical educators need to be aware of various influences on student ratings when developing data collection instruments and interpreting evaluation results. More research into the reliability and validity of overall course ratings as typically used in the evaluation of undergraduate medical education is warranted.

Diagnostic value of PSA and AP tests for the detection of spermatozoa in postmortem swabs from the genital and anal region in males

The aim of this study was to clarify whether positive results for prostate-specific antigen (PSA) and acid phosphatase (AP) occur in postmortem swabs from the genito-anal region in males (n = 80; 4 regions) and females (n = 20; 3 regions) and to calculate the positive predictive value (PPV) concerning the presence of spermatozoa. In male subjects, the highest incidence of positive test results was found in urethral swabs (PSA 76%, AP 71%) and the lowest frequencies appeared in perianal and rectal swabs (15-20%). Microscopic evaluation for spermatozoa was positive between 39% in urethral swabs and 1% in rectal swabs. PPV regarding positive identification of spermatozoa was 33.3% for PSA and 31.5% for AP. The combination of both tests yielded a PPV of 38.2%. In female cases, no spermatozoa were identified, and one case was PSA- and AP-positive in perianal swabs. Our findings indicate that PSA and AP tests are of limited value for the postmortem detection of spermatozoa in male subjects.

Smoking cessation education and training in U.K. medical schools: a national survey

INTRODUCTION

Smoking cessation is one of the most cost-effective of all health interventions. Physicians are in a strong position to encourage smokers to make a quit attempt and to help them achieve long-term abstinence. Formal teaching on tobacco-related disease, the evidence base of smoking cessation, and practical skills training regarding cessation advice and counseling are therefore important parts of undergraduate medical education. A survey of U.K. medical schools conducted 11 years ago revealed substantial deficits in the curricular coverage of these topics. This study aimed at establishing whether the situation has improved since then.

METHODS

In 2013, all U.K. medical schools were invited to participate in an online survey of their curricular coverage of tobacco addiction and smoking cessation.

RESULTS

Of the 33 medical schools, 22 (67%) schools responded. Health effects of smoking were addressed in more than 90% of curricula, and factual knowledge on nicotine addiction and withdrawal symptoms was covered in 50% of curricula. Only 1 in 3 medical schools offered practical skills training in artificial (i.e., role play) or clinical settings, and 50% of schools did not address smoking in summative assessments.

CONCLUSIONS

Practical skills training regarding cessation counseling is insufficient at most U.K. medical schools and may have become worse during the last 11 years. Increased curricular coverage-including summative assessments-of these topics would ensure that future physicians are adequately equipped to encourage and support effective evidence-based quit attempts in their patients.

Assessment of two different types of bias affecting the results of outcome-based evaluation in undergraduate medical education

BACKGROUND

Estimating learning outcome from comparative student self-ratings is a reliable and valid method to identify specific strengths and shortcomings in undergraduate medical curricula. However, requiring students to complete two evaluation forms (i.e. one before and one after teaching) might adversely affect response rates. Alternatively, students could be asked to rate their initial performance level retrospectively. This approach might threaten the validity of results due to response shift or effort justification bias.

METHODS

Two consecutive cohorts of medical students enrolled in a six-week cardio-respiratory module were enrolled in this study. In both cohorts, performance gain was estimated for 33 specific learning objectives. In the first cohort, outcomes calculated from ratings provided before (pretest) and after (posttest) teaching were compared to outcomes derived from comparative self-ratings collected after teaching only (thentest and posttest). In the second cohort, only thentests and posttests were used to calculate outcomes, but data collection tools differed with regard to item presentation. In one group, thentest and posttest ratings were obtained sequentially on separate forms while in the other, both ratings were obtained simultaneously for each learning objective.

RESULTS

Using thentest ratings to calculate performance gain produced slightly higher values than using true pretest ratings. Direct comparison of then- and posttest ratings also yielded slightly higher performance gain than sequential ratings, but this effect was negligibly small.

CONCLUSIONS

Given the small effect sizes, using thentests appears to be equivalent to using true pretest ratings. Item presentation in the posttest does not significantly impact on results.

Evaluation of medical research performance--position paper of the Association of the Scientific Medical Societies in Germany (AWMF)

Objective: The evaluation of medical research performance is a key prerequisite for the systematic advancement of medical faculties, research foci, academic departments, and individual scientists’ careers. However, it is often based on vaguely defined aims and questionable methods and can thereby lead to unwanted regulatory effects. The current paper aims at defining the position of German academic medicine toward the aims, methods, and consequences of its evaluation.

Methods: During the Berlin Forum of the Association of the Scientific Medical Societies in Germany (AWMF) held on 18 October 2013, international experts presented data on methods for evaluating medical research performance. Subsequent discussions among representatives of relevant scientific organizations and within three ad-hoc writing groups led to a first draft of this article. Further discussions within the AWMF Committee for Evaluation of Performance in Research and Teaching and the AWMF Executive Board resulted in the final consented version presented here.

Results: The AWMF recommends modifications to the current system of evaluating medical research performance. Evaluations should follow clearly defined and communicated aims and consist of both summative and formative components. Informed peer reviews are valuable but feasible in longer time intervals only. They can be complemented by objective indicators. However, the Journal Impact Factor is not an appropriate measure for evaluating individual publications or their authors. The scientific “impact” rather requires multidimensional evaluation. Indicators of potential relevance in this context may include, e.g., normalized citation rates of scientific publications, other forms of reception by the scientific community and the public, and activities in scientific organizations, research synthesis and science communication. In addition, differentiated recommendations are made for evaluating the acquisition of third-party funds and the promotion of junior scientists.

Conclusions: With the explicit recommendations presented in the current position paper, the AWMF suggests enhancements to the practice of evaluating medical research performance by faculties, ministries and research funding organizations.

Does research into medical education on tobacco and alcohol get the respect it deserves?

Implementation of effective clinical interventions as part of routine medical care to combat problem drinking and tobacco use is recognized to be low. Insufficient training is an important barrier preventing physicians from delivering interventions recommended in current guidelines. High-impact medical journals publish almost nothing on how to address this. If they start to take an interest perhaps we will see more progress and many thousands of lives saved at minimal cost.

A systematic review of studies assessing the association between adherence to smoking cessation medication and treatment success

AIMS

Lack of adherence to smoking cessation medication regimens is assumed to play a significant role in limiting their effectiveness. This study aimed to assess evidence for this assumption.

METHODS

A systematic search was conducted, supplemented by expert consultation, of papers reporting on randomized trials and observational studies examining the association between adherence to cessation medication and the success of quit attempts. To rule out reverse causality, only studies where adherence was assessed prior to relapse were included. Five studies met the inclusion criteria and results were extracted independently by two researchers. Heterogeneity between studies precluded a pooled analysis of the data.

RESULTS

Studies varied widely with regard to both the definition of adherence and outcome measures. The included studies only addressed adherence to nicotine replacement therapy. One study of lozenge use found that the amount of medication used between 1 and 2 weeks after the quit date predicted abstinence at 6 weeks [adjusted odds ratio (OR) for 'high' versus 'low' lozenge use 1.25; 95% confidence interval (CI) = 1.05-1.50; P < 0.02]. Similarly, one study found a significant impact of oral nicotine consumption during the first week on abstinence at 4 weeks (adjusted OR per additional mg/day = 1.05; CI = 1.01-1.10). Another study found that participants using nicotine replacement therapy for at least 5 weeks were significantly more likely to self-report continuous abstinence at 6 months. The remaining two studies failed to find a significant effect of treatment duration on outcome at 1 and 2 years but had very low power to detect such an effect.

CONCLUSIONS

There is modest evidence to support the assumption that lack of adherence to nicotine replacement therapy regimens undermines effectiveness in clinical studies.

Should we pay the student? A randomised trial of financial incentives in medical education

BACKGROUND

Financial incentives are effective in moderating physician and patient behaviour, but they have not been studied in the context of medical education.

AIM

This study assessed whether financial incentives can motivate students to acquire electrocardiogram (ECG) interpretation skills.

METHODS

Students enrolled for a cardio-respiratory teaching module (n = 121) were randomised to an intervention (financial incentive) or a control (book voucher raffle) condition. All students took three validated exams of ECG interpretation skills (at module entry, module exit and seven weeks later). Only the exit exam was financially incentivised in the intervention group. The primary outcome was the proportion of students who correctly identified ≥60% of clinically important diagnoses in the exit exam.

RESULTS

Financial incentives more than doubled the odds of correctly identifying ≥60% of diagnoses in the exit exam (adjusted odds ratio 2.44, 95% confidence interval 1.05-5.67) and significantly increased student learning time. However, there was no significant effect on performance levels in the retention exam.

CONCLUSIONS

Financial incentives increase reported learning time and examination results in the short-term. The lack of a sustained effect on performance suggests that financial incentives may foster a superficial or strategic rather than a deep approach to learning.

Increase in medical knowledge during the final year of undergraduate medical education in Germany

AIMS

In Germany, the final year of undergraduate medical education ('practice year') consists of three 16-week clinical attachments, two of which are internal medicine and surgery. Students can choose a specific specialty for their third 16-week attachment. Practice year students do not receive specific teaching to prepare them for the National Licensing Examination. It is unknown whether knowledge levels increase during this year. This study aimed at assessing knowledge at the beginning and the end of the final year of medical school.

METHODS

Three hundred pre-selected United States Medical Licensing Examination type items from ten medical disciplines were reviewed by ten recent medical graduates from the Netherlands and Germany. The resulting test included 150 items and was taken by 77 and 79 final year medical students from Göttingen and Hamburg at the beginning and the end of their practice year, respectively.

RESULTS

Cronbach's α of the pre- and post-test was 0.75 and 0.68, respectively. Mean percent scores in the pre- and post-test were 63.9±6.9 and 69.4±5.7, respectively (p<0.001; effect size calculated as Cohen's d: 0.87). In individual students, post-test scores were particularly high for items related to their specific chosen specialty.

CONCLUSION

The knowledge test used in this study provides a suitable external tool to assess progress of undergraduate medical students in their knowledge during the practice year. The pre-test may be used to guide individual learning behaviour during this final year of undergraduate education.

German medical students' beliefs about the effectiveness of different methods of stopping smoking

INTRODUCTION

In many countries, smoking cessation interventions are not routinely delivered as recommended in national and international guidelines. This may be because of incorrect beliefs about their effectiveness. This study assessed which cessation methods are believed to be effective by medical students in different years of undergraduate education as well as predictors of correct beliefs about effectiveness.

METHODS

In this cross-sectional study, undergraduate students from 27 German medical schools were invited to complete a survey addressing demographic characteristics, smoking status, self-rated knowledge of health consequences, and treatment options for smoking and beliefs about the effectiveness of 8 different methods to achieve long-term smoking cessation. Predictors of beliefs were identified by means of multilevel modeling.

RESULTS

A total of 19,526 students completed the survey. Students greatly overestimated the effectiveness of unaided quitting, and differences between years of undergraduate education were small. In the final year, 51% of students wrongly believed that willpower alone was more effective than a comprehensive group cessation program, including nicotine replacement therapy. Multilevel modeling revealed that having never smoked, supporting public smoking bans, and recalling theoretical training in smoking cessation were associated with correct beliefs.

CONCLUSIONS

A considerable proportion of German medical students believe that willpower alone is more effective than comprehensive treatment programs to support a quit attempt.

Future physicians and tobacco: an online survey of the habits, beliefs and knowledge base of medical students at a Canadian University

Background

Little is known about the knowledge and attitudes towards tobacco use among medical students in Canada. Our objectives were to estimate the prevalence of tobacco use among medical students, assess their perceived level of education about tobacco addiction management and their preparedness to address tobacco use with their future patients.

Methods

A cross-sectional online survey was administered to University of Alberta undergraduate medical school trainees. The 32-question survey addressed student demographics, tobacco use, knowledge and attitudes around tobacco and waterpipe smoking, tobacco education received in medical school, as well as knowledge and competency regarding tobacco cessation interventions.

Results

Of 681 polled students, 301 completed the survey. Current (defined as “use within the last 30 days”) cigarette, cigar/cigarillo and waterpipe smoking prevalence was 3.3%, 6% and 6%, respectively. One third of the respondents had ever smoked a cigarette, but 41% had tried cigars/cigarillos and 40% had smoked a waterpipe at some time in the past. Students reported moderate levels of education on a variety of tobacco-related subjects but were well-informed on the role of tobacco in disease causation. The majority of students in their final two years of training felt competent to provide tobacco cessation interventions, but only 10% definitively agreed that they had received enough training in this area.

Conclusions

Waterpipe exposure/current use was surprisingly high among this sample of medical students, a population well educated about the role of tobacco in disease causation. The majority of respondents appeared to be adequately prepared to manage tobacco addiction but education could be improved, particularly training in behavioral modification techniques used in tobacco use cessation.

Prevalence and characteristics of smokers interested in internet-based smoking cessation interventions: cross-sectional findings from a national household survey

Background

An accurate and up-to-date estimate of the potential reach of Internet-based smoking cessation interventions (ISCIs) would improve calculations of impact while an understanding of the characteristics of potential users would facilitate the design of interventions.

Objective

This study reports the prevalence and the sociodemographic, smoking, and Internet-use characteristics of smokers interested in using ISCIs in a nationally representative sample.

Methods

Data were collected using cross-sectional household surveys of representative samples of adults in England. Interest in trying an Internet site or “app” that was proven to help with stopping smoking was assessed in 1128 adult smokers in addition to sociodemographic characteristics, dependence, motivation to quit, previous attempts to quit smoking, Internet and handheld computer access, and recent types of information searched online.

Results

Of a representative sample of current smokers, 46.6% (95% CI 43.5%-49.6%) were interested in using an Internet-based smoking cessation intervention. In contrast, only 0.3% (95% CI 0%-0.7%) of smokers reported having used such an intervention to support their most recent quit attempt within the past year. After adjusting for all other background characteristics, interested smokers were younger (OR=0.98, 95% CI 0.97-0.99), reported stronger urges (OR=1.29, 95% CI 1.10-1.51), were more motivated to quit within 3 months (OR=2.16, 95% CI 1.54-3.02), and were more likely to have made a quit attempt in the past year (OR=1.76, 95% CI 1.30-2.37), access the Internet at least weekly (OR=2.17, 95% CI 1.40-3.36), have handheld computer access (OR=1.65, 95% CI 1.22-2.24), and have used the Internet to search for online smoking cessation information or support in past 3 months (OR=2.82, 95% CI 1.20-6.62). There was no association with social grade.

Conclusions

Almost half of all smokers in England are interested in using online smoking cessation interventions, yet fewer than 1% have used them to support a quit attempt in the past year. Interest is not associated with social grade but is associated with being younger, more highly motivated, more cigarette dependent, having attempted to quit recently, having regular Internet and handheld computer access, and having recently searched for online smoking cessation information and support.

Estimating learning outcomes from pre- and posttest student self-assessments: a longitudinal study

PURPOSE

Learning outcome is an important measure for overall teaching quality and should be addressed by comprehensive evaluation tools. The authors evaluated the validity of a novel evaluation tool based on student self-assessments, which may help identify specific strengths and weaknesses of a particular course.

METHOD

In 2011, the authors asked 145 fourth-year students at Göttingen Medical School to self-assess their knowledge on 33 specific learning objectives in a pretest and posttest as part of a cardiorespiratory module. The authors compared performance gain calculated from self-assessments with performance gain derived from formative examinations that were closely matched to these 33 learning objectives.

RESULTS

Eighty-three students (57.2%) completed the assessment. There was good agreement between performance gain derived from subjective data and performance gain derived from objective examinations (Pearson r=0.78; P<.0001) on the group level. The association between the two measures was much weaker when data were analyzed on the individual level. Further analysis determined a quality cutoff for performance gain derived from aggregated student self-assessments. When using this cutoff, the evaluation tool was highly sensitive in identifying specific learning objectives with favorable or suboptimal objective performance gains.

CONCLUSIONS

The tool is easy to implement, takes initial performance levels into account, and does not require extensive pre-post testing. By providing valid estimates of actual performance gain obtained during a teaching module, it may assist medical teachers in identifying strengths and weaknesses of a particular course on the level of specific learning objectives.

German medical students' beliefs about how best to treat alcohol use disorder

BACKGROUND/AIMS

A minority of German medical students believe they know how to support smokers willing to quit. This paper examined whether the same would be true for treating alcohol use disorder (AUD), and individual factors associated with incorrect beliefs about the effectiveness of methods to treat AUD.

METHODS

In this cross-sectional study, 19,526 undergraduate students from 27 German medical schools completed a survey addressing beliefs about the effectiveness of different methods of overcoming AUD. Beliefs about AUD treatment effectiveness were compared across the 5 years of undergraduate education and predictors identified by means of multiple linear regression.

RESULTS

Even in the fifth year, 28.1% (95% CI: 26.5-29.7) of students believed that willpower alone was more effective for overcoming AUD than a comprehensive treatment program. The only significant predictor of this belief was a similar belief for stopping smoking.

CONCLUSION

Our results indicate that a considerable proportion of German medical students overestimate the effectiveness of willpower to treat smoking and AUD. The addictive nature of these disorders needs to be stressed during undergraduate medical education to ensure that future physicians will be able and motivated to support patients in their quit attempts.

[Do final-year medical students know enough about the treatment of alcohol use disorders and smoking?]

BACKGROUND AND OBJECTIVE

Smoking and alcohol use disorders (AUD) are associated with significant morbidity and mortality in Germany. However, it has recently been shown that German medical students in years 1 to 5 do not feel competent to treat patients who are addicted to tobacco or alcohol. This study examined whether these deficits are also prevalent in students in the final (sixth) year of training.

METHODS

Students enrolled in the final year at University Medical Centre Hamburg-Eppendorf were invited to complete a questionnaire assessing smoking status and self-reported knowledge of health consequences of and treatment options for AUD and smoking as well as arterial hypertension and diabetes mellitus. Students were also asked to provide effectiveness estimates for different methods to treat AUD and smoking.

RESULTS

A total of 228 out of 345 students participated in the survey (response rate 66 %). Smoking prevalence was 24 %. Approximately 90 % of students believed they knew how to treat arterial hypertension and diabetes mellitus, but less than a third thought they knew how to treat smokers and patients with AUD. Effectiveness ratings of treatments for the two addictive disorders revealed severe misconceptions.

CONCLUSION

The deficits in undergraduate medical education regarding the treatment of addictive disorders reported for students from years 1 to 5 extend to students in the sixth year. Just before graduation, students still have severe knowledge gaps. In order to prevent tobacco- and alcohol-related deaths, medical school curricula need to be urgently improved.

German medical students lack knowledge of how to treat smoking and problem drinking

AIM

To assess the extent of undergraduate medical training on alcohol use disorders (AUD) and smoking, and medical students' perceived knowledge regarding consequences of, and treatment options for, these disorders compared with other chronic conditions.

DESIGN

Cross-sectional survey assessing teaching and perceived knowledge of health consequences and treatment options for AUD and smoking compared with diabetes and hypertension.

SETTING

Medical schools in Germany.

PARTICIPANTS

Twenty-five of 36 medical school offices (response rate 69.4%) and 19 526 of 39 358 students from 27 medical schools (response rate 49.6%).

MEASUREMENT

Medical schools were asked to provide information on curricular coverage of the four conditions. Students reported their year of study and perceived knowledge about the consequences of all four disorders and perceived knowledge of treatment options.

FINDINGS

Courses time-tabled approximately half as many teaching hours on AUD and tobacco as on diabetes or hypertension. Final-year students reported high levels of knowledge of consequences of all four conditions and how to treat diabetes and hypertension, but only 20% believed they knew how to treat alcohol use disorders or smoking.

CONCLUSIONS

Curriculum coverage in German medical schools of alcohol use disorders and smoking is half that of diabetes and hypertension, and in the final year of their undergraduate training most students reported inadequate knowledge of how to intervene to address them.

Structured smoking cessation training for health professionals on cardiology wards: a prospective study

BACKGROUND

Smoking is a major cardiovascular risk factor, and smoking cessation is imperative for patients hospitalized with a cardiovascular event. This study aimed to evaluate a systems-based approach to helping hospitalized smokers quit and to identify implementation barriers.

DESIGN

Prospective intervention study followed by qualitative analysis of staff interviews.

METHODS

The prospective intervention study assessed the effects of implementing standard operating procedures (SOPs) for the provision of counselling and pharmacotherapy to smokers admitted to cardiology wards on counselling frequency. In addition, a qualitative analysis of staff interviews was undertaken to examine determinants of physician and nurse behaviour; this sought to understand barriers in terms of motivation, capability, and/or opportunity.

RESULTS

A total of 150 smoking patients were included in the study (75 before and 75 after SOP implementation). Before the implementation of SOPs, the proportion of patients reporting to have received cessation counselling from physicians and nurses was 6.7% and 1.3%, respectively. Following SOP implementation, these proportions increased to 38.7% (p < 0.001) and 2.7% (p = 0.56), respectively. Qualitative analysis revealed that lack of motivation, e.g. role incongruence, appeared to be a major barrier.

CONCLUSIONS

Introduction of a set of standard operating procedures for smoking cessation advice was effective with physicians but not nurses. Analysis of barriers to implementation highlighted lack of motivation rather than capability or opportunity as a major factor that would need to be addressed.