Prioritizing areas for faculty development of clinical teachers by using student evaluations for evidence-based decisions

Comparison of the Perspectives of Medical Students and Residents on the Surgery Learning Environment

BACKGROUND

The learning environment plays a critical role in learners' satisfaction and outcomes. However, we often lack insight into learners' perceptions and assessments of these environments. It can be difficult to discern learners' expectations, making their input critical. When medical students and surgery residents are asked to evaluate their teachers, what do they focus on?

MATERIALS AND METHODS

Open-ended comments from medical students' evaluations of residents and attending surgeons and from residents' evaluations of attendings during the 2016-2017 academic year were analyzed. Content analysis was used, and codes derived from the data. A matrix of theme by learner role was created to distinguish differences between medical student and resident learners. Subthemes were grouped based on similarity into high-order themes.

RESULTS

Two overarching themes were Creating a positive environment for learning by modeling professional behaviors and Intentionally engaging learners in training and educational opportunities. Medical students and residents made similar comments for the subthemes of appropriate demeanor, tone and dialog, respect, effective direct instruction, feedback, debriefing, giving appropriate levels of autonomy, and their expectations as team members on a service. Differences existed in the subthemes of punctuality, using evidence, clinical knowledge, efficiency, direct interactions with patients, learning outcomes, and career decisions.

CONCLUSIONS

Faculty development efforts should target professional communication, execution of teaching skills, and relationships among surgeons, other providers, and patients. Attendings should make efforts to discuss their approach to clinical decision making and patient interactions and help residents and medical students voice their opinions and questions through trusting adult learner-teacher relationships.

Residents as teachers: Near peer learning in clinical work settings: AMEE Guide No. 106

This AMEE Guide provides a framework to guide medical educators engaged in the design and implementation of "Resident as Teacher" programs. The suggested approaches are based on established models of program development: the Program Logic model to guide program design, the Dundee three-circle model to inform a systematic approach to planning educational content and the Kirkpatrick pyramid, which forms the backbone of program evaluation. The Guide provides an overview of Resident as Teacher curricula, their benefits and impact, from existing literature supplemented by insights from the authors' own experiences, all of whom are engaged in teaching initiatives at their own institutions. A conceptual description of the Program Logic model is provided, a model that highlights an outcomes-based curricular design. Examples of activities under each step of this model are described, which would allow educational leaders to structure their own program based on the scope, context, institutional needs and resources available. Emphasis is placed on a modular curricular format to not only enhance the teaching skills of residents, but also enable development of future career educators, scholars and leaders. Application of the Dundee three-circle model is illustrated to allow for a flexible curricular design that can cater to varying levels of educational needs and interests. In addition, practical advice is provided on robust assessment of outcomes, both assessment of participants and program evaluation. Finally, the authors highlight the need for congruence between the formal and hidden curriculum through explicit recognition of the value of teaching by institutions, support for development of teaching programs, encouragement of evidence-based approach to education and rewards for all levels of teachers.

Assessing the quality of clinical teaching: a preliminary study

OBJECTIVES

Evaluations in the clinical arena are fraught with problems. Current assessments of clinical teaching typically measure attributes of clinical teachers in overly broad terms, are often subjective and often succumb to the halo effect. This is in contradistinction to measurements of lectures, workshops or online educational content, which can more readily be assessed using objective criteria. As a result, clinical evaluations are often insufficient to provide focused feedback, guide faculty development or identify specific areas for clinical teachers to implement change and improvement. The aim of our study was to offset these limitations.

METHODS

We developed a structured, 15-item objective structured clinical examination (OSCE)-type checklist of discrete teaching behaviours intended to be: (i) observable; (ii) applicable to multiple disciplines, and (iii) reliably identifiable. Our goal was to test and utilise this checklist as an objective assessment of clinical teaching across a range of in-patient teaching rounds experiences. During 2007-2008, pairs of external raters on two separate occasions observed nine attending physicians during actual in-patient paediatrics and internal medicine ward rounds at a large, academic medical centre. Observers documented the extent to which specific teaching behaviours did or did not occur.

RESULTS

The internal consistency of the 15-item checklist was good (alpha = 0.85). A two-facet, partially nested G study found the generalisability of ratings to be generally acceptable, but inter-rater reliability varied greatly between occasions and across individual checklist items.

CONCLUSIONS

Despite attempts to identify discrete and observable target behaviours, placing observers on rounds to detect these behaviours may not be as straightforward as it would seem. Clinical teaching may be a more inherently subjective process, based on different teaching styles of faculty staff. However, a set of objective checklist items to be completed by trained observers on teaching rounds holds promise as a potentially viable means of identifying strengths and weaknesses of clinical instruction. Further research is needed to define what constitutes quality clinical teaching, as well as the most reliable method for assessing it.

A systematic review of resident-as-teacher programmes

CONTEXT

Residents in all disciplines serve as clinical teachers for medical students. Since the 1970s, there has been increasing evidence to demonstrate that residents wish to teach and that they respond positively to formal teacher training. Effective resident-as-teacher (RaT) programmes have resulted in improved resident teaching skills. Current evidence, however, is not clear about the specific features of an effective RaT programme.

OBJECTIVES

This study was performed in order to investigate the effectiveness of RaT programmes on resident teaching abilities and to identify the features that ensure success. Methods of assessment used to ascertain the effectiveness of RaT programmes are also explored.

METHODS

The literature search covered the period between 1971 and 2008. Articles focusing on improving resident teaching skills were included. Each study was reviewed by two reviewers and data were collected using a standard abstraction summary sheet. Study outcomes were graded according to a modified Kirkpatrick's model of educational outcomes.

RESULTS

Twenty-nine studies met review inclusion criteria. Interventions included workshops, seminars, lectures and teaching retreats. Twenty-six studies used a pre- and post-intervention outcome comparison method. Subjective outcome measures included resident self-evaluation of teaching skills or evaluation by medical students, peers and faculty members. Objective outcome measures included written tests, evaluation of teaching performance by independent raters and utilisation of objective structured teaching examinations. One study objectively measured learning outcomes at the level of medical students, utilising the results of an objective structured clinical examination. Overall resident satisfaction with RaT programmes was high. Participants reported positive changes in attitudes towards teaching. Participant knowledge of educational principles improved. Study methodologies allowed for significant risks of bias.

CONCLUSIONS

More rigorous study designs and the use of objective outcome measures are needed to ascertain the true effectiveness of RaT programmes. Future research should focus on determining the impact of RaT programmes on learning achievement at the level of medical students.

The development of a participant questionnaire to assess continuing medical education presentations

OBJECTIVE

Feedback on presentation skills is important for developing skilled educators, but often this feedback is based on evaluation tools that have been developed with little concern for psychometric issues or for how the information will be used for feedback. The purpose of this study was to develop a reliable participant questionnaire to assess the quality of continuing medical education (CME) presentations and to provide presenters with feedback.

DESIGN

The questionnaire was developed using an iterative approach, with doctors as raters, and tested during a variety of CME presentations. The resulting questionnaire consists of 9 items rated on a 7-point Likert scale. The psychometric analysis reported in this paper was completed using data from grand rounds presentations.

RESULTS

Psychometric analysis, based on 319 evaluations from 17 presentations (average of 19 evaluations/presentation), revealed a high level of reliability (0.91), indicating that the items met a reasonable standard and that the raters were discriminating between the quality of the presentations adequately.

CONCLUSION

This 9-item, participant questionnaire provides a reliable measure of the quality of CME presentations, while also providing presenters with useful feedback. Further studies will investigate if this instrument can be used to assess other CME formats and how best to provide feedback to presenters.