Web-based peer evaluation by medical students

Web-assisted assessment of professional behaviour in problem-based learning: more feedback, yet no qualitative improvement?

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category 'Dealing with oneself' consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase.

Comparison of student examiner to faculty examiner scoring and feedback in an OSCE

OBJECTIVES

to help reduce pressure on faculty staff, medical students have been used as raters in objective structured clinical examinations (OSCEs). There are few studies regarding their ability to complete checklists and global rating scales, and a paucity of data on their ability to provide feedback to junior colleagues. The objectives of this study were: (i) to compare expert faculty examiner (FE) and student-examiner (SE) assessment of students' (candidates') performances on a formative OSCE; (ii) to assess SE feedback provided to candidates, and (iii) to seek opinion regarding acceptability from all participants.

METHODS

year 2 medical students (candidates, n = 66) participated in a nine-station formative OSCE. Year 4 students (n = 27) acted as SEs and teaching doctors (n = 27) served as FEs. In each station, SEs and FEs independently scored the candidates using checklists and global rating scales. The SEs provided feedback to candidates after each encounter. The FEs evaluated SEs on the feedback provided using a standardised rating scale (1 = strongly disagree, 5 = strongly agree) for several categories, according to whether the feedback was: balanced; specific; accurate; appropriate; professional, and similar to feedback the FE would have provided. All participants completed questionnaires exploring perceptions and acceptability.

RESULTS

there was a high correlation on the checklist items between raters on each station, ranging from 0.56 to 0.86. Correlations on the global rating for each station ranged from 0.23 to 0.78. Faculty examiners rated SE feedback highly, with mean scores ranging from 4.02 to 4.44 for all categories. There was a high degree of acceptability on the part of candidates and examiners.

CONCLUSIONS

student-examiners appear to be a viable alternative to FEs in a formative OSCE in terms of their ability to both complete checklists and provide feedback.

Teaching and learning professional behavior in practice

This paper is the fourth article in a series on Professionalism and provides an overview of current methods used for teaching and learning about professionalism. The questions "whether" and "how" professionalism can be placed in the formal medical school curricula are addressed, and the informal learning related to professionalism reviewed.

Teaching and Assessing Professionalism in Ophthalmology Residency Training Programs

The Accreditation Council for Graduate Medical Education (ACGME) has mandated that all residency training programs teach and assess new competencies including professionalism. This article reviews the literature on medical professionalism, describes good practices gleaned from published works, and proposes an implementation matrix of specific tools for teaching and assessing professionalism in ophthalmology residency. Professionalism requirements have been defined by the ACGME, subspecialty organizations, and other certifying and credentialing organizations. Teaching, role modeling, and assessing the competency of professionalism are important tasks in managing the ACGME mandate. Future work should focus on the field testing of tools for validity, reliability, feasibility, and cost-effectiveness.

Fostering professionalism in medical education: a call for improved assessment and meaningful incentives

Increasing attention has been focused on developing professionalism in medical school graduates. Unfortunately, the culture of academic medical centers and the behaviors that faculty model are often incongruent with our image of professionalism. The need for improved role modeling, better assessment of student behavior, and focused faculty development is reviewed. We propose that the incentive structure be adjusted to reward professional behavior in both students and faculty. The third-year medicine clerkship provides an ideal opportunity for clinician-educators to play a leading role in evaluating, rewarding, and ultimately fostering professionalism in medical school graduates.

Assessing residents' competency in care management: report of a consensus conference

BACKGROUND

Residency programs must prepare physicians to practice in the current health care environment. This mandate is reflected in 3 of the 6 competency domains now required by the Accreditation Council for Graduate Medical Education: systems-based practice, interpersonal skills and communication, and practice-based learning and improvement.

SUMMARY

An invitational conference was convened, with experts in clinical practice, managed care administration, and education to identify and recommend optimal and promising assessment methods for 4 target areas: physician-patient communication, ethics, teamwork and collaboration, and practice management. Working in small groups, participants considered a range of resident assessment methods and identified current or future methods for each area, based on reliability, validity, use of behaviorally oriented outcomes, feasibility, and cost. Preferred methods of assessment varied by domain and include written examinations, computer-based patient management problems, standardized patients, objective structured clinical examinations, portfolios, 360-degree evaluations, and patient satisfaction surveys.

CONCLUSIONS

The use of several practical, scientifically sound, and specific methods for assessing residents' competency in care management are recommended. Assessment instruments will need to be flexible enough to adapt to the rapid changes in the health care delivery system and terminology.

An Internet-based tool for evaluating third-year medical student performance

BACKGROUND

Shortened non-primary care medical school clerkships have increased time pressures for accurate assessment of student knowledge, skills, and attitudes. Paper-based student evaluations suffer from low response rates, inefficient data acquisition and analysis, and difficulty obtaining input from multiple evaluators. This project describes the development of a Web-based model for evaluating third-year medical student performance, improving evaluation response rates, and including multiple evaluators' input.

METHODS

A secure Web-based system was designed to maintain evaluation data (11-item competency-based evaluations, oral examinations, National Board of Medical Examiners surgery test, and objective structured clinical examination) for the third-year surgery clerkship. Historical response rate, completion time, and administrative effort data were compared with data obtained using the Web-based model.

RESULTS

Faculty response rates improved from 71.3% to 89.9%, with response times decreased from 28.0 +/- 3.0 to 9.0 +/- 0.7 days using the Web-based model. Administrative time requirements decreased from 5 days to 2 hours per rotation, and manual data entry, analysis, and reporting were eliminated through e-mail evaluator notification, direct data entry, and real-time analysis. Evaluator satisfaction was subjectively higher using the Web-based model.

CONCLUSIONS

The Web-based 360-degree evaluation model improves third-year medical student assessment by including residents, reducing time and cost, and by providing a faster, more inclusive, and efficient evaluation.