Exams at medical faculties - quality, responsibility and perspectives

[The examination mix at the Medical University Vienna]

The present work describes a curriculum design with the instrument of an examination mix for the study of medicine at the medical University of Vienna. The fields of application of special examination formats as well as the advantages and disadvantages of the individual examination forms are presented. Types of summative written examinations, assessment of practical skills and abilities (OSCE), oral-practical examinations, as well as formative examinations are illustrated. Studies show that repeated testing leads to higher learning gains. Therefore, the challenge is to develop suitable methods for the continuous assessment of the learning progress in the study and to apply accordingly. Thus, in addition to the written summative exams, more and more oral and practical forms of examinations in courses with continuous assessment character should be used.

[Examinations while studying medicine - more than simply grades]

Assessment drives learning. Examinations need to be aligned primarily with learning objectives, as well as teaching and assessment methods of the courses on offer. In doing so, various examination instruments are required to measure on levels of competency that build on one another. An appropriate mix is essential to reflect the variety of learning outcomes of a chosen curriculum. Furthermore, examinations also possess the characteristics of evaluation: They reflect the knowledge and abilities of students and assess the teaching at a defined location. Digital examinations in the form of multiple-choice-question (MCQ) testing enable a higher degree of automation and accelerate the processes of creation, implementation, and evaluation of the examination results. Thus, they enjoy increasing popularity, provided that the technical requirements for large semester cohorts are met. Shifting examination processes to computers or tablets entails not only a wealth of new challenges but also opportunities.

Assessment formats in dental medicine: An overview

AIM

At the annual meeting of German dentists in Frankfurt am Main in 2013, the Working Group for the Advancement of Dental Education (AKWLZ) initiated an interdisciplinary working group to address assessments in dental education. This paper presents an overview of the current work being done by this working group, some of whose members are also actively involved in the German Association for Medical Education's (GMA) working group for dental education. The aim is to present a summary of the current state of research on this topic for all those who participate in the design, administration and evaluation of university-specific assessments in dentistry.

METHOD

Based on systematic literature research, the testing scenarios listed in the National Competency-based Catalogue of Learning Objectives (NKLZ) have been compiled and presented in tables according to assessment value.

RESULTS

Different assessment scenarios are described briefly in table form addressing validity (V), reliability (R), acceptance (A), cost (C), feasibility (F), and the influence on teaching and learning (EI) as presented in the current literature. Infoboxes were deliberately chosen to allow readers quick access to the information and to facilitate comparisons between the various assessment formats. Following each description is a list summarizing the uses in dental and medical education.

CONCLUSION

This overview provides a summary of competency-based testing formats. It is meant to have a formative effect on dental and medical schools and provide support for developing workplace-based strategies in dental education for learning, teaching and testing in the future.

The Final Oral/Practical State Examination at Freiburg Medical Faculty in 2012--Analysis of grading to test quality assurance

AIM

The aim of this study is to analyze the grades given for the oral/practical part of the German State Examination at the Medical Faculty of Freiburg. We examined whether or not the grades given for the written and the oral/practical examinations correlated and if differences in grading between the Freiburg University Medical Center (UMC) and the other teaching hospitals could be found. In order to improve the quality of the state examination, the medical school has been offering standardized training for examiners for several years. We evaluated whether or not trained and untrained examiners differed in their grading of the exam and how these differences have changed over time.

METHODS

The results of the 2012 spring and fall exams were analyzed (N=315). The relevant data set was made available to us by the Baden-Württemberg Examination Office (Landesprüfungsamt). The data were analyzed by means of descriptive and inferential statistics.

RESULTS

We observed a correlation of ρ=0.460** between the grades for the written and the oral/practical exams. The UMC and the teaching hospitals did not differ significantly in their grade distributions. Compared to untrained examiners, trained ones assigned the grade of "very good" less often. Furthermore, they displayed a significantly higher variance in the grades given (p=0.007, phi=0.165). This effect is stronger when concentrating specifically on those examiners who took part in the training less than a year before.

CONCLUSION

The results of this study suggest that the standardized training for examiners at the Medical Faculty of Freiburg is effective for quality assurance. As a consequence, more examiners should be motivated to take part in the training.

Impact of the Medical Faculty on Study Success in Freiburg: Results from Graduate Surveys

AIM

Using the data from graduate surveys, this study aims to analyze which factors related to teaching and learning at the Freiburg Faculty of Medicine can influence study success.

BACKGROUND

Study success and the factors influencing it have long been the subject of investigation, with study success being measured in terms of easily quantifiable indicators (final grades, student satisfaction, etc.). In recent years, it has also frequently been assessed in terms of graduate competency levels. Graduate surveys are considered suitable instruments for measuring these dimensions of study success.

METHOD

Data from three Freiburg graduate surveys conducted one and a half years after graduation were drawn upon for the analysis. Study success was operationalized using four indicators: results on the written section of the M2 exam, self-assessment of medical expertise and scientific expertise, and student satisfaction. Using multiple regression analyses, the predictive power was calculated for selected variables, also measured by the graduate surveys, for the different study success indicators.

RESULTS

It was possible to identify models that contribute slightly or moderately to the prediction of study success. The score earned on the university entrance qualification demonstrated itself to be the strongest predictor for forecasting the M2 written exam: R(2) is between 0.08 and 0.22 for the three surveys. Different variables specific to degree program structure and teaching are helpful for predicting medical expertise (R(2)=0.04-0.32) and student satisfaction (R(2)=0.12-0.35). The two variables, structure and curricular sequencing of the degree program and combination of theory and practice, show themselves to be significant, sample-invariant predictors (β-weight(Structure)=0.21-0.58, β-weight(Combination)=0.27-0.56). For scientific expertise, no sample-independent predictors could be determined.

CONCLUSION

Factors describing teaching hardly provide any assistance when predicting the written M2 exam score, which makes sense to the extent that teaching goes far beyond the heavily knowledge-based content of the written M2 exam. The lack of predictability for scientific expertise is most likely explained in that these have been only rarely included in the curriculum and often inexplicitly so. The variable combination of theory and practice appears to be significant for imparting medical expertise and the development of student satisfaction. The extent to which these relationships are practically relevant needs to be explored in further studies. A specific limitation is that the measurement of expertise and skill is based solely on self-assessments.

Quality management of clinical-practical instruction for Practical Year medical students in Germany - proposal for a catalogue of criteria from the German Society of Medical Education

Objectives: Amended in 2013, the current version of the German Medical Licensure Regulation contains structural specifications that are also required of non-university institutions involved in Practical Year clinical training. The criteria are worded in relatively general terms. Furthermore, not all of the structural specifications can be readily applied to every subject area. In order to ensure commensurability in Practical Year instruction in Germany, not least in light of recently introduced Practical Year mobility, it is necessary to define consistent quality criteria for Practical Year training. The authors therefore propose a catalogue of criteria for the quality management process in Practical Year instruction facilities.

Methods: In January 2014, the board of directors of the German Society for Medical Education decided to establish a committee comprised of representatives from various German medical faculties. In a process similar to the Delphi methodology, the group developed criteria for structure, process and outcome quality in Practical Year training in Germany.

Results: The criteria developed for structure, process and outcome quality apply to Practical Year training in academic teaching hospitals and university medical centres. Furthermore, modalities for review are proposed.

Conclusions: The present catalogue of criteria is intended to contribute to the formation of a basis for the most consistent quality standards possible for Practical Year instruction in Germany.

Students benefit from developing their own emergency medicine OSCE stations: a comparative study using the matched-pair method

BACKGROUND

Students can improve the learning process by developing their own multiple choice questions. If a similar effect occurred when creating OSCE (objective structured clinical examination) stations by themselves it could be beneficial to involve them in the development of OSCE stations. This study investigates the effect of students developing emergency medicine OSCE stations on their test performance.

METHOD

In the 2011/12 winter semester, an emergency medicine OSCE was held for the first time at the Faculty of Medicine at the University of Leipzig. When preparing for the OSCE, 13 students (the intervention group) developed and tested emergency medicine examination stations as a learning experience. Their subsequent OSCE performance was compared to that of 13 other students (the control group), who were parallelized in terms of age, gender, semester and level of previous knowledge using the matched-pair method. In addition, both groups were compared to 20 students who tested the OSCE prior to regular emergency medicine training (test OSCE group).

RESULTS

There were no differences between the three groups regarding age (24.3 ± 2.6; 24.2 ± 3.4 and 24 ± 2.3 years) or previous knowledge (29.3 ± 3.4; 29.3 ± 3.2 and 28.9 ± 4.7 points in the multiple choice [MC] exam in emergency medicine). Merely the gender distribution differed (8 female and 5 male students in the intervention and control group vs. 3 males and 17 females in the test OSCE group).In the exam OSCE, participants in the intervention group scored 233.4 ± 6.3 points (mean ± SD) compared to 223.8 ± 9.2 points (p < 0.01) in the control group. Cohen's effect size was d = 1.24. The students of the test OSCE group scored 223.2 ± 13.4 points.

CONCLUSIONS

Students who actively develop OSCE stations when preparing for an emergency medicine OSCE achieve better exam results.