Impact of peer feedback on the performance of lecturers in emergency medicine: a prospective observational study

Does peer feedback for teaching GPs improve student evaluation of general practice attachments? A pre-post analysis

Objectives: The extent of university teaching in general practice is increasing and is in part realised with attachments in resident general practices. The selection and quality management of these teaching practices pose challenges for general practice institutes; appropriate instruments are required. The question of the present study is whether the student evaluation of an attachment in previously poorly evaluated practices improves after teaching physicians have received feedback from a colleague. Methods: Students in study years 1, 2, 3 and 5 evaluated their experiences in general practice attachments with two 4-point items (professional competence and recommendation for other students). Particularly poorly evaluated teaching practices were identified. A practising physician with experience in teaching and research conducted a personal feedback of the evaluation results with these (peer feedback), mainly in the form of individual discussions in the practice (peer visit). After this intervention, further attachments took place in these practices. The influence of the intervention (pre/post) on student evaluations was calculated in generalised estimating equations (cluster variable practice). Results: Of 264 teaching practices, 83 had a suboptimal rating. Of these, 27 practices with particularly negative ratings were selected for the intervention, of which 24 got the intervention so far. There were no post-evaluations for 5 of these practices, so that data from 19 practices (n=9 male teaching physicians, n=10 female teaching physicians) were included in the present evaluation. The evaluations of these practices were significantly more positive after the intervention (by n=78 students) than before (by n=82 students): odds ratio 1.20 (95% confidence interval 1.10-1.31; p<.001). Conclusion: The results suggest that university institutes of general practice can improve student evaluation of their teaching practices via individual collegial feedback.

Coaching educators: Impact of a novel national faculty development program for didactic presentation skills

BACKGROUND

Didactic lectures remain common in medical education. Many faculty physicians do not receive formal training on public presentations or leading instructional sessions. Coaching has emerged in medical education with the potential to positively impact skills. We sought to evaluate a novel, national faculty peer-coaching program created to improve lecture presentation skills and foster career development.

METHODS

This was a mixed-methods study of participant and faculty perceptions after completing the Council of Residency Directors in Emergency Medicine Academy Coaching Program. Participants completed an online evaluative survey consisting of multiple choice and Likert-type items. Program coaches participated in semistructured interviews. Descriptive statistics were reported for survey data. Thematic qualitative analysis by two independent reviewers was performed on interview data.

RESULTS

During 2012 to 2017, a total of 30 participants and 11 coaches from 37 residency programs across the United States engaged in the program. Twenty-four (80%) participants completed the survey. Eight (73%) coaches participated in semistructured interviews. Data were collected between October and December 2018. The mean ± SD numbers of national presentations participants had given before and after the coaching program were 6.92 ± 7.68 and 16.42 ± 15.43, respectively. Since their coaching, most participants (87.5%) have been invited to give a lecture at another institution. Many participants felt that the program improved their lecture evaluations, public speaking, ability to engage an audience, and professional development. Almost all (92%) would recommend the program to a colleague. The coaches perceived multiple benefits including improved skills, self-reflection, networking, career advancement, and personal fulfillment. Suggestions for improvement included improved administrative processes, more clear expectations, increased marketing, and increased participant and coach engagement.

CONCLUSION

Participants and coaches perceived multiple benefits from this novel, national faculty coaching program. With identification of the success, challenges, and suggestions for improvement, others may benefit as they develop coaching programs in medical education.

Do they teach what they need to? An analysis of the impact of curriculum mapping on the learning objectives taught in a lecture series in surgery

Introduction: Curriculum mapping shows concordances and differences between the intended and the taught curriculum. To our knowledge, no previous studies describe the effects that this mapping has on the curriculum. The aim of the present study is to map the content of a lecture series in surgery to the National Catalogue of Learning Objectives in Surgery and analyze the effects this mapping has on the content of the following lecture series. Methods: All lecturers in the lecture series were directly observed by a minimum of two reviewers and learning objectives and the level of competence were documented. After the lecture series, the results were visualized within the catalog of learning objectives and were sent to the lecturers. In the following lecture series, learning objectives were documented correspondingly. Results: In the first lecture series, 47% of the learning objectives were taught. After the mapping, the number of learning objectives that were taught increased to 59% (p < 0.001). The increase was found in all surgical disciplines and in all levels of competences without any changes in the average duration of the lectures. Conclusions: The presented method for mapping a curriculum effectively increased the number of taught learning objectives without requiring longer lecture durations.

The Council of Emergency Medicine Residency Directors Academy for Scholarship Coaching Program: Addressing the Needs of Academic Emergency Medicine Educators

Introduction

Didactic lectures remain fundamental in academic medicine; however, many faculty physicians do not receive formal training in instructional delivery. In order to design a program to instill and enhance lecture skills in academic emergency medicine (EM) physicians we must first understand the gap between the current and ideal states.

Methods

In 2012 the Council of Emergency Medicine Residency Directors (CORD) Academy for Scholarship designed a novel coaching program to improve teaching skills and foster career development for medical educators based on literature review and known teaching observation programs. In order to inform the refinement of the program, we performed a needs assessment of participants. Participants’ needs and prior teaching experiences were gathered from self-reflection forms completed prior to engaging in the coaching program. Two independent reviewers qualitatively analyzed data using a thematic approach.

Results

We analyzed data from 12 self-reflection forms. Thematic saturation was reached after nine forms. Overall inter-rater agreement was 91.5%. We categorized emerging themes into three domains: participant strengths and weaknesses; prior feedback with attempts to improve; and areas of desired mentorship. Several overlapping themes and subthemes emerged including factors pertaining to the lecturer, the audience/learner, and the content/delivery.

Conclusion

This study identified several areas of need from EM educators regarding lecture skills. These results may inform faculty development efforts in this area. The authors employed a three-phase, novel, national coaching program to meet these needs.

Coaching the Debriefer: Peer Coaching to Improve Debriefing Quality in Simulation Programs

STATEMENT

Formal faculty development programs for simulation educators are costly and time-consuming. Peer coaching integrated into the teaching flow can enhance an educator's debriefing skills. We provide a practical guide for the who, what, when, where, why, and how of peer coaching for debriefing in simulation-based education. Peer coaching offers advantages such as psychological safety and team building, and it can benefit both the educator who is receiving feedback and the coach who is providing it. A feedback form for effective peer coaching includes the following: (1) psychological safety, (2) framework, (3) method/strategy, (4) content, (5) learner centeredness, (6) co-facilitation, (7) time management, (8) difficult situations, (9) debriefing adjuncts, and (10) individual style and experience. Institutional backing of peer coaching programs can facilitate implementation and sustainability. Program leaders should communicate the need and benefits, establish program goals, and provide assessment tools, training, structure, and evaluation to optimize chances of success.

The effect of written standardized feedback on the structure and quality of surgical lectures: A prospective cohort study

BACKGROUND

Lectures remain an important teaching method to present and structure knowledge to many students concurrently. Adequate measures are necessary to maintain the quality of the lectures. The aim of this study was to determine the impact on the lecture quality using written structured feedback and to compare the ratings of surgical lectures between students and surgical peers.

METHODS

Prospective analysis of two consecutive surgical lecture series for undergraduate students at Goethe-University Medical School was performed before and after evaluation of the lecturers via independent written feedback from trained undergraduate students and surgeons. The 22-item feedback instrument covered three areas of performance: content, visualization, and delivery. Additional suggestions for improvement were provided from both students and surgical peers who anonymously attended the lectures. The lecturers, experienced surgeons, as well as the student and peer raters were blinded in terms of the aim and content of the study. Their response to the feedback was collected using a web-based 13-item questionnaire. The Kendall's-W coefficient was computed to calculate inter-rater reliability (IRR). Differences between ratings before and after feedback were analyzed using Student's t-test for dependent samples. The Kolmogorov-Smirnov-test was used for independent samples.

RESULTS

A total of 22 lectures from a possible 32 given by 13 lecturers were included and analyzed by at least three surgeons and two students. There were significant improvements in overall score as well as in the details of 9 of the 13 items were found. The average inter-rater reliability was 0.71. There were no differences in the ratings as a function of the rater's level of expertise (peers vs. students). We found that 13/23 lecturers (56.5%) answered the questionnaire, and 92% strongly agreed that the written feedback was useful. 76.9% of the lecturers revised their lecture based on the written feedback requiring on average 112.5 min (range from 20 to 300 min).

CONCLUSIONS

Overall, this study indicates that structured written feedback provided by trained peers and students that is subsequently discussed by the lecturers concerned is a highly effective and efficient method to improve aspects of lecturing. We anticipate that structured written feedback by trained students that is discussed by the lecturers concerned will improve lecturing.

Frame-of-Reference Training: Establishing Reliable Assessment of Teaching Effectiveness

INTRODUCTION

Frame-of-reference (FOR) training has been used successfully to teach faculty how to produce accurate and reliable workplace-based ratings when assessing a performance. We engaged 21 Harvard Medical School faculty members in our pilot and implementation studies to determine the effectiveness of using FOR training to assess health professionals' teaching performances.

METHODS

All faculty were novices at rating their peers' teaching effectiveness. Before FOR training, we asked participants to evaluate a recorded lecture using a criterion-based peer assessment of medical lecturing instrument. At the start of training, we discussed the instrument and emphasized its precise behavioral standards. During training, participants practiced rating lectures and received immediate feedback on how well they categorized and scored performances as compared with expert-derived scores of the same lectures. At the conclusion of the training, we asked participants to rate a post-training recorded lecture to determine agreement with the experts' scores.

RESULTS

Participants and experts had greater rating agreement for the post-training lecture compared with the pretraining lecture. Through this investigation, we determined that FOR training is a feasible method to teach faculty how to accurately and reliably assess medical lectures.

DISCUSSION

Medical school instructors and continuing education presenters should have the opportunity to be observed and receive feedback from trained peer observers. Our results show that it is possible to use FOR rater training to teach peer observers how to accurately rate medical lectures. The process is time efficient and offers the prospect for assessment and feedback beyond traditional learner evaluation of instruction.