Comparison of resident and medical student evaluation of faculty teaching

Better Together: Development and Implementation of Fellow Group Evaluations of Faculty

Background

High-quality trainee evaluations of faculty are essential for meaningful faculty development and for improving the clinical learning environment. However, concerns about anonymity can limit usefulness of trainee evaluations, particularly in smaller programs, such as subspecialty fellowships.

Objective

To develop and implement a fellow-driven group evaluation process to enhance trainee confidentiality and generate high-quality feedback for pulmonary and critical care medicine faculty.

Methods

A novel process was developed for faculty evaluation and feedback consisting of quarterly, structured, fellow-led group evaluation sessions focused on collecting confidential, behaviorally oriented, actionable feedback for faculty. Upper-year fellow moderators utilized a standard format to structure discussion, generating strengths and areas for growth for each faculty member while explicitly asking for input from fellows with divergent perspectives. Moderators compiled anonymized session notes for the program director, who delivered feedback to individual faculty. After the first six sessions, an electronic survey was distributed to assess fellow perceptions of the group evaluation model.

Results

Thirty-seven faculty members were evaluated in 11 group sessions over 42 months. Fellows rated group-generated feedback as more confidential, more specific, more accurate, more efficient, more actionable, and less biased when compared with individual written evaluations (P < 0.01 for all categories).

Conclusion

The authors successfully developed and implemented a process for fellow-led group evaluation of faculty, designed to facilitate fellow confidentiality and enrich the quality of feedback. Fellows preferred the group evaluation process and perceived group-generated feedback more favorably compared with individual written evaluations.

Surgeons have an opportunity to improve teaching quality through feedback provision

BACKGROUND

Medical student evaluations of faculty are increasingly incorporated into promotion and tenure decisions, making it imperative to understand learner perceptions of quality teaching. Prior work has shown that students value faculty responsiveness in the form of feedback, but faculty and students differ in their perceptions of what constitutes sufficient feedback. The innovative minute feedback system (MFS) can quantify responsiveness to students' feedback requests. This study assessed how feedback provision via MFS impacts teaching quality scores.

MATERIALS AND METHODS

This retrospective observational study compared average faculty teaching quality scores with faculty's percentage response to student feedback requests via the MFS. The data were generated from the core surgical clerkship for third-year medical students at the University of Michigan Medical School. The relationship between average teaching quality scores and response percentage was assessed by weighted regression analysis.

RESULTS

Two hundred thirty-seven medical students requested feedback via MFS, and 104 faculty were evaluated on teaching quality. The mean faculty feedback response percentage was 55.78%. The mean teaching quality score was 4.27 on a scale of 1 to 5. Teaching quality score was significantly correlated with response percentage (P < 0.001); for every 10% increase in response percentage, average teaching quality score improved by 0.075. Average teaching quality score was not significantly associated with response time (P = 0.158), gender (P = 0.407), or surgical service (P = 0.498).

CONCLUSIONS

Medical students consider responsiveness to feedback requests an important component of quality teaching. Furthermore, faculty development focused on efficient and practical feedback strategies may have the added benefit of improving their teaching quality.

A Faculty Toolkit for Formative Assessment in Pharmacy Education

This paper aims to increase understanding and appreciation of formative assessment and its role in improving student outcomes and the instructional process, while educating faculty on formative techniques readily adaptable to various educational settings. Included are a definition of formative assessment and the distinction between formative and summative assessment. Various formative assessment strategies to evaluate student learning in classroom, laboratory, experiential, and interprofessional education settings are discussed. The role of reflective writing and portfolios, as well as the role of technology in formative assessment, are described. The paper also offers advice for formative assessment of faculty teaching. In conclusion, the authors emphasize the importance of creating a culture of assessment that embraces the concept of 360-degree assessment in both the development of a student's ability to demonstrate achievement of educational outcomes and a faculty member's ability to become an effective educator.

Confounding factors in using upward feedback to assess the quality of medical training: a systematic review

PURPOSE

Upward feedback is becoming more widely used in medical training as a means of quality control. Multiple biases exist, thus the accuracy of upward feedback is debatable. This study aims to identify factors that could influence upward feedback, especially in medical training.

METHODS

A systematic review using a structured search strategy was performed. Thirty-five databases were searched. Results were reviewed and relevant abstracts were shortlisted. All studies in English, both medical and non-medical literature, were included. A simple pro-forma was used initially to identify the pertinent areas of upward feedback, so that a focused pro-forma could be designed for data extraction.

RESULTS

A total of 204 articles were reviewed. Most studies on upward feedback bias were evaluative studies and only covered Kirkpatrick level 1-reaction. Most studies evaluated trainers or training, were used for formative purposes and presented quantitative data. Accountability and confidentiality were the most common overt biases, whereas method of feedback was the most commonly implied bias within articles.

CONCLUSION

Although different types of bias do exist, upward feedback does have a role in evaluating medical training. Accountability and confidentiality were the most common biases. Further research is required to evaluate which types of bias are associated with specific survey characteristics and which are potentially modifiable.

A novel resident-as-teacher training program to improve and evaluate obstetrics and gynecology resident teaching skills

BACKGROUND

Residents play a significant role in teaching, but formal training, feedback, and evaluation are needed.

AIMS

Our aims were to assess resident teaching skills in the resident-as-teacher program, quantify correlations of faculty evaluations with resident self-evaluations, compare resident-as-teacher evaluations with clinical evaluations, and evaluate the resident-as-teacher program.

METHOD

The resident-as-teacher training program is a simulated, videotaped teaching encounter with a trained medical student and standardized teaching evaluation tool. Evaluations from the resident-as-teacher training program were compared to evaluations of resident teaching done by faculty, residents, and medical students from the clinical setting.

RESULTS

Faculty evaluation of resident teaching skills in the resident-as-teacher program showed a mean total score of 4.5 ± 0.5 with statistically significant correlations between faculty assessment and resident self-evaluations (r = 0.47; p < 0.001). However, resident self-evaluation of teaching skill was lower than faculty evaluation (mean difference: 0.4; 95% CI 0.3-0.6). When compared to the clinical setting, resident-as-teacher evaluations were significantly correlated with faculty and resident evaluations, but not medical student evaluations. Evaluations from both the resident-as-teacher program and the clinical setting improved with duration of residency.

CONCLUSIONS

The resident-as-teacher program provides a method to train, give feedback, and evaluate resident teaching.

Academic advancement of clinician educators: why is it so difficult?

The role of clinician educators (CEs) in institutions and medical centres continues to be vital without any doubt. Although there has been more than a century since Sir William Osler established the role of the CE and the tradition by encouraging bed-side teaching, there is still a lack of consensus on the attributes that define a 'clinician-educator'. The concept of a superior clinician who is also a dedicated teacher seems to fit the description of a CE but most often seems insufficient to support the CE's academic advancement.

Evaluation of physiology lectures conducted by students: comparison between evaluation by staff and students

As a peer-assisted learning process, minilectures on physiology were conducted by students. During this process, students lecture to their colleagues in the presence of faculty staff members. These lectures were evaluated by faculty staff and students simultaneously. The aim of this study was to compare feedback from faculty members and students on 66 minilectures conducted by students. Their perception of different qualities of lecture was assessed using a questionnaire. There were significant correlations between students and faculty members for many qualities of the lecture, including the speed of the lecture, retaining attention, clear introduction, and the overall quality of the lecture. However, ratings for gesture, eye contact, language usage, illustration usage, audiovisuals, voice usage, and important points stressed were significantly different between students and faculty members. Multiple regression analysis was performed to assess the degree of effect of different aspects of a lecture on its overall quality. Aspects such as gesture, eye contact, and language usage showed very low β-values, suggesting a poor contribution of these factors to the overall quality of the lecture for both students and faculty members. The speed of the lecture, retaining attention, and clear introduction were qualities that faculty members and students rated equally, and these were the main contributors to the overall quality of the lecture. Awareness about the possible discrepancy between ratings given by faculty members and students may be important when interpreting the evaluation results of formal lectures by these two groups.

The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis

BACKGROUND

Assessment of clinical teaching by learners is of value to teachers, department heads, and program directors, and must be comprehensive and feasible.

AIMS

To review published evaluation instruments with psychometric evaluations and to develop and psychometrically evaluate an instrument for assessing clinical teaching with linkages to the CanMEDS roles.

METHOD

We developed a 19-item questionnaire to reflect 10 domains relevant to teaching and the CanMEDS roles. A total of 317 medical learners assessed 170 instructors. Fourteen (4.4 %) clinical clerks, 229 (72.3%) residents, and 53 (16.7%) fellows assessed 170 instructors. Twenty-one (6.6%) did not specify their position.

RESULTS

A mean number of eight raters assessed each instructor. The internal consistency reliability of the 19-item instrument was Cronbach's α = 0.95. The generalizability coefficient (Ep(2)) analysis indicated that the raters achieved Ep(2) of 0.95. The factor analysis showed three factors that accounted for 67.97% of the total variance. The three factors together, with the variance accounted for and their internal consistency reliability, are teaching skills (variance = 53.25s%; Cronbach's α = 0.92), Patient interaction (variance = 8.56%; Cronbach's α = 0.91), and professionalism (variance = 6.16%; Cronbach's α = 0.86). The three factors are intercorrelated (correlations = 0.48, 0.58, 0.46; p < 0.01).

CONCLUSION

It is feasible to assess clinical teaching with the 19-item instrument that has demonstrated evidence of both validity and reliability.

Are successful resident clinicians good teachers? Multidimensional resident assessment

BACKGROUND

GME training programs must demonstrate residents are prepared as both teachers and clinicians. We examined the relationship between faculty evaluations of residents' clinical performance and student evaluations of residents' teaching performance.

METHOD

Concordance tables for mean ratings and qualitative analysis of comments among 95 residents receiving evaluations by 267 faculty and 106 students.

RESULTS

A total of 88% of residents received concordant ratings from faculty and students. When discordant, faculty gave higher ratings (P = .003). Ninety and ninety-one percent of faculty and student comments exhibited a positive tone.

CONCLUSIONS

Faculty and students tended to agree on the successful and unsuccessful residents, supporting the validity of rating scale evaluations. Future research might focus on how to tailor resident remediation when discordant ratings or comments occur.

Examination of a clinical teaching effectiveness instrument used for summative faculty assessment

This study explores whether a clinical teaching effectiveness (CTE) instrument provides valid scores for summative faculty assessment. The sample included all CTE instruments (n = 10,087) that learners (N = 1,194) completed to assess clinical teachers (N = 872) during 1 academic year. The authors investigated response processes (e.g., missing data, straight-line responses, level of learner), internal structure (e.g., confirmatory and exploratory factor analysis), teaching ratings by learner group (medical student or resident), and relation to other variables (e.g., correlation with global rating). Response processes identified a high prevalence of straight-line responses (same rating across all items) and differential patterns of missing data by learner group. Medical students rated their teachers higher than residents, and CTE scores had different factor structures depending on learner group. High correlation coefficients of CTE items with a single rating of overall teaching performance suggest that learners consider global performance when assessing clinical teaching performance.

Medical student and academic staff perceptions of role models: an analytical cross-sectional study

BACKGROUND

This study explored the associations between the perceptions of students and the perceptions of academic staff about the characteristics of clinical lecturers at the Department of Internal Medicine at Kerman University of Medical Sciences (KUMS). It also assessed what characteristics constitute a 'role model' from the point of view of students and staff.

METHODS

Staff and students were questioned about the characteristics of their colleagues and lecturers, respectively. They were asked about 15 characteristics under four headings: personality, teaching skill, group working and overall performance as a role model. Associations between lecturers' characteristics were explored using Pearson correlation and characteristics were allocated into groups by partition cluster method. In addition, predictors of being a valuable lecturer were assessed using logistic regression analysis.

RESULTS

Based on staff responses, the strongest association observed was between honesty and being respectful (r = 0.93, p < 0.0001). Based on student responses, the strongest association observed was between being professional and honesty (r = 0.98, p < 0.0001). None of the correlations between student and staff perceptions were significant for any characteristic. Two groups were recognized among the characteristics. group one contained those characteristics which were related to the lecturer's activity; while the second group contained characteristics that were related to the personality or teaching performance of the lecturer. The predictors of lecturer as 'role model' (i.e., perceptions of students) consisted mostly of characteristics from the first group, while the predictors of a 'role model' by fellow academic staff consisted of characteristics that were in both groups.

CONCLUSION

These findings showed considerable differences between the perceptions of students about their lecturers when compared with perceptions of staff about their colleagues. Students were more concerned with the personality of their lecturers, while staff also considered their ideas and behaviors. This suggests that a more comprehensive assessment of a lecturer's performance could be obtained by taking into account feedback from both students and colleagues.

Does academic advancement impact teaching performance of surgical faculty?

BACKGROUND

An important reason for young surgeons entering academic practice is to educate trainees. As clinical and administrative responsibility increases, teaching time is decreased. We attempted to determine if the teaching performance of surgeons declines with career advancement.

METHODS

Between July 1998 and June 2002, all faculty at our institution were evaluated by medical students on the surgical clerkship. Surgeons were scored on clinical teaching (CLINIC), operating room teaching (OR), and overall teaching (ALL) with the use of a 4-point scale. Surgeons were grouped by years in practice; group scores were compared with ANOVA.

RESULTS

A total of 6345 evaluations were completed on 74 academic surgeons. Junior surgeons (< or = 5 years) performed better in operating room teaching (P < .001), clinical teaching (P<.001), and overall teaching (P < .001) compared with those in practice more than 5 years. When junior surgeons were compared with the most senior faculty (> or = 15 years), the difference in all categories was even greater (P < .001). Tenure status had no relationship to teaching ability.

CONCLUSIONS

Junior faculty surgeons were perceived to be more effective teachers when compared to senior faculty. With career advancement, faculty may devote less energy to teaching and become less effective. Therefore, continued emphasis should be placed on strengthening education skills throughout the surgical career.

Clinician-teachers' self-assessments versus learners' perceptions

Understanding how clinician-teachers' self-assessments compare to learners' impressions can serve to help educators place each of these evaluations in the appropriate context. Past participants of the Johns Hopkins Faculty Development Program and other physician-teachers were surveyed in 2002 regarding their teaching skills and behaviors. We surveyed their learners to compare teacher and learner assessments of teaching proficiency, behaviors, enjoyment, and career satisfaction. In each area, learners' ratings were statistically significantly higher than their teachers' self-ratings. Though it is unclear whether teachers' or learners' assessments are a more accurate reflection of the truth, the more positive learner ratings should promote self-confidence in clinician-educators regarding their teaching abilities.

Evaluations of clinical faculty: the impact of level of learner and time of year

BACKGROUND

Because of the importance of faculty evaluation data for faculty development, promotion, reappointment, and remuneration, the quantity and characteristics of evaluation data need study.

PURPOSES

To examine differences in evaluation ratings according to level of learner (student vs. resident) and time of year.

METHODS

Retrospective analyses of 1,281 faculty evaluations provided for 132 faculty over a 12-month period by students and residents on inpatient medicine rotations.

RESULTS

Faculty had an average of eight evaluations over a year. Students gave more favorable ratings than residents. Over the year, students became less critical and residents became more critical. Winners of teaching awards had better evaluations than other faculty. Standard errors around mean ratings for an individual faculty member were approximately 0.25 on a 4-point scale.

CONCLUSIONS

The profile and precision of assessments for an individual will vary based on the number, timing, and source of ratings. Decisions based on similar data should be made with caution.