Limitations of student evaluations of curriculum

How was the intern year?: self and clinical assessment of four cohorts, from two medical curricula

BACKGROUND

Problem-based curricula have provoked controversy amongst educators and students regarding outcome in medical graduates, supporting the need for longitudinal evaluation of curriculum change. As part of a longitudinal evaluation program at the University of Adelaide, a mixed method approach was used to compare the graduate outcomes of two curriculum cohorts: traditional lecture-based 'old' and problem-based 'new' learning.

METHODS

Graduates were asked to self-assess preparedness for hospital practice and consent to a comparative analysis of their work-place based assessments from their intern year. Comparative data were extracted from 692 work-place based assessments for 124 doctors who graduated from the University of Adelaide Medical School between 2003 and 2006.

RESULTS

Self-assessment: Overall, graduates of the lecture-based curriculum rated the medical program significantly higher than graduates of the problem-based curriculum. However, there was no significant difference between the two curriculum cohorts with respect to their preparedness in 13 clinical skills. There were however, two areas where the cohorts rated their preparedness in the 13 broad practitioner competencies as significantly different: problem-based graduates rated themselves as better prepared in their 'awareness of legal and ethical issues' and the lecture-based graduates rated themselves better prepared in their 'understanding of disease processes'.Work-place based assessment: There were no significant differences between the two curriculum cohorts for 'Appropriate Level of Competence' and 'Overall Appraisal'. Of the 14 work-place based assessment skills assessed for competence, no significant difference was found between the cohorts.

CONCLUSIONS

The differences in the perceived preparedness for hospital practice of two curriculum cohorts do not reflect the work-place based assessments of their competence as interns. No significant difference was found between the two cohorts in relation to their knowledge and clinical skills. However results suggest a trend in 'communication with peers and colleagues in other disciplines' (χ2 (3, N = 596) =13.10, p = 0.056) that requires further exploration. In addition we have learned that student confidence in a new curriculum may impact on their self-perception of preparedness, while not affecting their actual competence.

The role of radiology in preclinical anatomy: a critical review of the past, present, and future

RATIONALE AND OBJECTIVES

Radiology has been an increasingly important component of preclinical anatomy instruction since the 1960s. The global status of medical imaging pedagogies and radiologists' roles in medical anatomy education is not well established but is important in determining the specialty's contribution to undergraduate medical education.

MATERIALS AND METHODS

PubMed was searched with various combinations of MeSH terms including "radiology," "undergraduate medical education," and "anatomy." Articles were reviewed for relevance, and referenced articles of possible relevance were hand-traced to ensure a wide capture of articles.

RESULTS

Although more medical schools around the world are using medical imaging to teach anatomy, some regions, such as the United States, show a decline in the proportion of imaging taught by radiologists. Lectures, small group discussions, and self-instruction remain the mainstay of current pedagogies and have witnessed dramatic changes over the past few decades with respect to the types of imaging used. Newer pedagogies use contextual and hands-on experiences to improve spatial and application principles. Qualitative and quantitative studies report somewhat mixed results of pedagogical efficacies but demonstrate generally high acceptance by students and instructors and often significant exam score improvement. Radiology as a specialty must overcome several challenges for it to become more involved in anatomy education, including teaching incentives and protected academic time.

CONCLUSIONS

As anatomy instruction and clinical medicine grow increasingly digital, it is ever more important that radiologists continue to develop new anatomy pedagogies and contribute to anatomy education in greater roles.

Student Course Evaluations: Common Themes across Courses and Years

Student course evaluations were analyzed for common themes across five different basic science, clinical, and innovative courses from the first and third years of medical school. Each course had both unique and common numerically scaled items including an overall quality rating item. A principal components analysis was conducted for each course to determine the items that loaded most heavily on the same component as the overall quality item. Across courses and years, the items that consistently loaded on the same component as the overall quality item were (1) administrative aspects including course organization, (2) clearly communicated goals and objectives, and (3) instructional staff responsiveness. These results concur with recent medical education literature in this area. Faculty interested in increasing student ratings of the overall quality of their courses might best attend primarily to carefully organizing course goals and objectives and clearly communicating them. The limitations of these conclusions are discussed.

Dedicated core clerkship in radiology for medical students development, implementation, evaluation, and comparison with distributed clerkship

RATIONALE AND OBJECTIVES

The authors performed this study to develop, implement, and evaluate a dedicated core clerkship in radiology for the required clinical clerkship year of medical school, and to compare it with the distributed core clerkship that it replaced.

MATERIALS AND METHODS

A dedicated 5-day clerkship was added to the clinical core year of medical school. The course offered a variety of learning experiences, including lectures, clinical observation, case discussions, and a team project. Learner achievement was measured by posttest and compared with a control group. Student satisfaction was determined by structured and unstructured surveys. Faculty comment was elicited by survey, and administrative staff perspective was established through interviews. The evaluation of the dedicated clerkship was compared with the distributed clerkship along the dimensions of learner achievement, student satisfaction, faculty comment, and administrative staff perspective.

RESULTS

The dedicated clerkship was developed and implemented successfully. Compared with the distributed clerkship, there was no significant difference in learner achievement or student satisfaction for the dedicated clerkship, but the dedicated clerkship was easier to conduct for faculty and administrative staff.

CONCLUSION

The dedicated clerkship was advantageous for faculty and administrative staff, whereas maintaining a comparable level of learner achievement and student satisfaction as the distributed clerkship.

Challenges Facing Radiology Educators

This article summarizes the current challenges faced by radiology educators. These are discussed under the headings of time available for education, financing education, what and how to teach, student evaluation, support infrastructure, and challenges to leadership.

Investigating the use of sampling for maximising the efficiency of student-generated faculty teaching evaluations

PURPOSE

Surveys of medical students are widely used to evaluate course content and faculty teaching within the medical school. Gathering information that accurately reflects student perceptions requires that students buy into the evaluation process and be willing to provide thoughtful responses to the teaching evaluation. To maintain student commitment, it is important that medical students are not overburdened with poorly planned evaluations. Sampling might decrease the number of evaluations required of students and might also reduce the proportion of non-responses and other forms of inattentive response biases.

METHODS

A sampling technique employed within a large medical lecture is described and evaluated. A generalisability study of the teacher evaluations is conducted.

RESULTS

A high response rate and high levels of reliability were obtained by sampling a small proportion of the total class. The largest source of error was related to rater and utilising sufficient numbers of student-raters is critical to achieving reliable results.

CONCLUSION

Sampling can reduce evaluation demands placed on students, and preserve reliability and increase the validity of mean evaluation scores. With computer presentation, efficient sampling techniques become practical and should be part of software packages used to present teacher evaluations.

Faculty and resident preference for two different forms of lecture evaluation

OBJECTIVE

This study was undertaken to compare the use of 2 different lecture evaluation forms developed for obstetrics and gynecology residents to evaluate core curriculum lectures given by faculty.

STUDY DESIGN

Content and delivery of faculty lectures were evaluated by residents using (1) a 10-question evaluation form that used a 5-point rating scale and (2) a checklist evaluation form that offered multiple options to improve lectures but provided no numerical score. Each form was used exclusively for 4 months. Faculty received feedback from both forms; then faculty and residents were surveyed regarding their preferences.

RESULTS

A total of 384 rating scale and 398 checklist evaluation forms were completed during the study period. Residents preferred to complete the rating scale evaluation forms (67%), whereas faculty preferred feedback from the checklist evaluation forms (62%). Faculty were more likely to plan changes to their lecture format with feedback from the checklist evaluation forms (75% vs 25%, P = .031).

CONCLUSION

Although devoid of structured positive feedback, the checklist lecture evaluation form was preferred by faculty.