Examination of a clinical teaching effectiveness instrument used for summative faculty assessment

Teaching quality in neurosurgery: quantitating outcomes over time

OBJECTIVE

High-quality neurosurgery resident training is essential to developing competent neurosurgeons. Validated formative tools to assess faculty teaching performance exist, but are not used widely among Accreditation Council for Graduate Medical Education (ACGME) residency programs in the United States. Furthermore, their longer-term impact on teaching performance improvement and educational outcomes remains unclear. The goal of this study was to assess the impact of implementing an evaluation system to provide faculty with feedback on teaching performance in a neurosurgery residency training program over a 4-year period.

METHODS

The authors performed a prospective cohort study in which a modified version of the System for Evaluation of Teaching Qualities (SETQ) instrument was administered to neurosurgical trainees in their department regularly every 6 months. The authors analyzed subscale score dynamics to identify the strongest correlates of faculty teaching performance improvement. ACGME program survey results and trainee performance on written board examinations were compared for the 3 years before and after SETQ implementation.

RESULTS

The overall response rate among trainees was 91.8%, with 1044 surveys completed for 41 faculty. Performance scores improved progressively from cycle 1 to cycle 6. The strongest correlate of overall performance was providing positive feedback to trainees. Compared to the 3 years prior, the 3 years following SETQ implementation saw significant increases in written board examination and ACGME resident survey scores compared to the national mean.

CONCLUSIONS

Implementation of SETQ was associated with significant improvements in faculty teaching performance as judged by trainees over a 4-year period, and guided curricular changes in the authors' training program that resulted in improved educational outcomes.

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review

PURPOSE

To collect and examine-using an argument-based validity approach-validity evidence of questionnaire-based tools used to assess physicians' clinical, teaching, and research performance.

METHOD

In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians' professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians' clinical, teaching, and research performance. Using Kane's validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications.

RESULTS

They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane's validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing.

CONCLUSIONS

Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians' clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

Supporting formative peer review of clinical teaching through a focus on process

BACKGROUND

The professional need for development of clinical faculty is clear. Previous scholarship provides insight into the formative potential of peer review in both didactic and experiential settings. Less information exists on a comprehensive peer review process (PRP) designed to support faculty change.

EDUCATIONAL ACTIVITY AND SETTING

A clinical faculty PRP was developed and implemented based on input from the literature, stakeholders, and field experts. The process included: 1) self-reflective pre-work, 2) a peer-observation component, 3) self-reflective post-work, and 4) creation of a specific action plan via meeting with an educational expert. The process was assessed by collecting evaluative data from peer reviewer and clinical faculty participants.

FINDINGS

Eight of 26 faculty members participated in a pilot of the PRP and formed four clinical faculty-peer dyads. When surveyed, all participants unanimously reported that they would participate in the PRP again. Aspects perceived among most helpful to clinical teaching included peer observation, self-reflection, and meeting with an educational expert. Challenges related to the process included anxiety of peer observation, burden of pre-work, and logistics of scheduling meetings.

DISCUSSION

While instruments are important in guiding and documenting the evaluation of clinical teaching during an observation period, this initiative focused on the process supporting the observation and evaluation, in order to optimize the formative feedback received by participating faculty and encourage professional development actions.

SUMMARY

A PRP that incorporates preparation, reflective practice, and a meeting with an educational expert may support meaningful faculty development in the area of clinical teaching.

Voluntary vs. compulsory student evaluation of clerkships: effect on validity and potential bias

BACKGROUND

Students evaluations of their learning experiences can provide a useful source of information about clerkship effectiveness in undergraduate medical education. However, low response rates in clerkship evaluation surveys remain an important limitation. This study examined the impact of increasing response rates using a compulsory approach on validity evidence.

METHODS

Data included 192 responses obtained voluntarily from 49 third-year students in 2014-2015, and 171 responses obtained compulsorily from 49 students in the first six months of the consecutive year at one medical school in Lebanon. Evidence supporting internal structure and response process validity was compared between the two administration modalities. The authors also tested for potential bias introduced by the use of the compulsory approach by examining students' responses to a sham item that was added to the last survey administration.

RESULTS

Response rates increased from 56% in the voluntary group to 100% in the compulsory group (P < 0.001). Students in both groups provided comparable clerkship rating except for one clerkship that received higher rating in the voluntary group (P = 0.02). Respondents in the voluntary group had higher academic performance compared to the compulsory group but this difference diminished when whole class grades were compared. Reliability of ratings was adequately high and comparable between the two consecutive years. Testing for non-response bias in the voluntary group showed that females were more frequent responders in two clerkships. Testing for authority-induced bias revealed that students might complete the evaluation randomly without attention to content.

CONCLUSIONS

While increasing response rates is often a policy requirement aimed to improve the credibility of ratings, using authority to enforce responses may not increase reliability and can raise concerns over the meaningfulness of the evaluation. Administrators are urged to consider not only response rates, but also representativeness and quality of responses in administering evaluation surveys.

Opportunities for the CTEI: disentangling frequency and quality in evaluating teaching behaviours

Students' perceptions of teaching quality are vital for quality assurance purposes. An increasingly used, department-independent instrument is the (Cleveland) clinical teaching effectiveness instrument (CTEI). Although the CTEI was developed carefully and its validity and reliability confirmed, we noted an opportunity for improvement given an intermingling in its rating scales: the labels of the answering scales refer to both frequency and quality of teaching behaviours. Our aim was to investigate whether frequency and quality scores on the CTEI items differed. A sample of 112 residents anonymously completed the CTEI with separate 5-point rating scales for frequency and quality. Differences between frequency and quality scores were analyzed using paired t tests. Quality was, on average, rated higher than frequency, with significant differences for ten out of 15 items. The mean scores differed significantly in favour of quality. As the effect size was large, the difference in mean scores was substantial. Since quality was generally rated higher than frequency, the authors recommend distinguishing frequency from quality. This distinction helps to obtain unambiguous outcomes, which may be conducive to providing concrete and accurate feedback, improving faculty development and making fair decisions concerning promotion, tenure or salary.

Mobile pediatric emergency response team: patient satisfaction during the novel H1N1 influenza outbreak

OBJECTIVES

The objective was to determine child caregiver satisfaction with a nontraditional pediatric emergency department (ED) venue during the 2009 novel H1N1 influenza outbreak.

METHODS

Between May 1 and 7, 2009, the Texas Children's Hospital (TCH) ED used a six-bed outdoor facility, the Mobile Pediatric Emergency Response Team (MPERT), to evaluate patients with suspected novel H1N1 influenza. Parents and caregivers of patients evaluated in the MPERT were surveyed by telephone using a validated questionnaire to evaluate satisfaction with the facility.

RESULTS

Of 353 patients, 155 caregivers (44%) completed questionnaires; 127 had wrong numbers, 71 did not answer, and 15 were on a no-call list. Survey responders felt that nurses and doctors explained concepts well (nurses 92%, doctors 94%), 91% felt TCH prepared them well for taking care of their children at home, 94% were satisfied with the medical care received, and 88% were not bothered by the outdoor setting. When asked to rate their MPERT experience on a scale of 0 (worst possible) to 10 (best possible), the median score was 9 (range 1 to 10).

CONCLUSIONS

The MPERT facility alleviated patient volume surge and potentially prevented transmission during H1N1 outbreak. While these were health care provider goals, caregiver expectations were also met. Caregivers perceived MPERT as an acceptable alternative to receiving care in the regular ED, felt that physicians and nurses communicated well, and felt that medical care was good to excellent. Use of the MPERT did not negatively affect overall caregiver satisfaction with TCH. These findings suggest that families of pediatric patients are amenable to nontraditional ED venues during periods of ED crowding.

The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis

BACKGROUND

Assessment of clinical teaching by learners is of value to teachers, department heads, and program directors, and must be comprehensive and feasible.

AIMS

To review published evaluation instruments with psychometric evaluations and to develop and psychometrically evaluate an instrument for assessing clinical teaching with linkages to the CanMEDS roles.

METHOD

We developed a 19-item questionnaire to reflect 10 domains relevant to teaching and the CanMEDS roles. A total of 317 medical learners assessed 170 instructors. Fourteen (4.4 %) clinical clerks, 229 (72.3%) residents, and 53 (16.7%) fellows assessed 170 instructors. Twenty-one (6.6%) did not specify their position.

RESULTS

A mean number of eight raters assessed each instructor. The internal consistency reliability of the 19-item instrument was Cronbach's α = 0.95. The generalizability coefficient (Ep(2)) analysis indicated that the raters achieved Ep(2) of 0.95. The factor analysis showed three factors that accounted for 67.97% of the total variance. The three factors together, with the variance accounted for and their internal consistency reliability, are teaching skills (variance = 53.25s%; Cronbach's α = 0.92), Patient interaction (variance = 8.56%; Cronbach's α = 0.91), and professionalism (variance = 6.16%; Cronbach's α = 0.86). The three factors are intercorrelated (correlations = 0.48, 0.58, 0.46; p < 0.01).

CONCLUSION

It is feasible to assess clinical teaching with the 19-item instrument that has demonstrated evidence of both validity and reliability.

Assessing the quality of clinical teachers: a systematic review of content and quality of questionnaires for assessing clinical teachers

BACKGROUND

Learning in a clinical environment differs from formal educational settings and provides specific challenges for clinicians who are teachers. Instruments that reflect these challenges are needed to identify the strengths and weaknesses of clinical teachers.

OBJECTIVE

To systematically review the content, validity, and aims of questionnaires used to assess clinical teachers.

DATA SOURCES

MEDLINE, EMBASE, PsycINFO and ERIC from 1976 up to March 2010.

REVIEW METHODS

The searches revealed 54 papers on 32 instruments. Data from these papers were documented by independent researchers, using a structured format that included content of the instrument, validation methods, aims of the instrument, and its setting.

RESULTS

Aspects covered by the instruments predominantly concerned the use of teaching strategies (included in 30 instruments), supporter role (29), role modeling (27), and feedback (26). Providing opportunities for clinical learning activities was included in 13 instruments. Most studies referred to literature on good clinical teaching, although they failed to provide a clear description of what constitutes a good clinical teacher. Instrument length varied from 1 to 58 items. Except for two instruments, all had to be completed by clerks/residents. Instruments served to provide formative feedback ( instruments) but were also used for resource allocation, promotion, and annual performance review (14 instruments). All but two studies reported on internal consistency and/or reliability; other aspects of validity were examined less frequently.

CONCLUSIONS

No instrument covered all relevant aspects of clinical teaching comprehensively. Validation of the instruments was often limited to assessment of internal consistency and reliability. Available instruments for assessing clinical teachers should be used carefully, especially for consequential decisions. There is a need for more valid comprehensive instruments.

Rubric evaluation of pediatric emergency medicine fellows

OBJECTIVES

To develop and validate a rubric assessment instrument for use by pediatric emergency medicine (PEM) faculty to evaluate PEM fellows and for fellows to use to self-assess.

METHODS

This is a prospective study at a PEM fellowship program. The assessment instrument was developed through a multistep process: (1) development of rubric format items, scaled on the modified Dreyfus model proficiency levels, corresponding to the 6 Accreditation Council for Graduate Medical Education core competencies; (2) determination of content and construct validity of the items through structured input and item refinement by subject matter experts and focus group review; (3) collection of data using a 61-item form; (4) evaluation of psychometrics; (5) selection of items for use in the final instrument.

RESULTS

A total of 261 evaluations were collected from 2006 to 2007; exploratory factor analysis yielded 5 factors with Eigenvalues >1.0; each contained ≥4 items, with factor loadings >0.4 corresponding with the following competencies: (1) medical knowledge and practice-based learning and improvement, (2) patient care and systems-based practice, (3) interpersonal skills, (4) communication skills, and (5) professionalism. Cronbach α for the final 53-item instrument was 0.989. There was also significant responsiveness of the tool to the year of training.

CONCLUSION

A substantively and statistically validated rubric evaluation of PEM fellows is a reliable tool for formative and summative evaluation.

Are online student evaluations of faculty influenced by the timing of evaluations?

Student evaluations of faculty are important components of the medical curriculum and faculty development. To improve the effectiveness and timeliness of student evaluations of faculty in the physiology course, we investigated whether evaluations submitted during the course differed from those submitted after completion of the course. A secure web-based system was developed to collect student evaluations that included numerical rankings (1-5) of faculty performance and a section for comments. The grades that students received in the course were added to the data, which were sorted according to the time of submission of the evaluations and analyzed by Pearson's correlation and Student's t-test. Only 26% of students elected to submit evaluations before completion of the course, and the average faculty ratings of these evaluations were highly correlated [r(14) = 0.91] with the evaluations submitted after completion of the course. Faculty evaluations were also significantly correlated with the previous year [r(14) = 0.88]. Concurrent evaluators provided more comments that were statistically longer and subjectively scored as more "substantive." Students who submitted their evaluations during the course and who included comments had significantly higher final grades in the course. In conclusion, the numeric ratings that faculty received were not influenced by the timing of student evaluations. However, students who submitted early evaluations tended to be more engaged as evidenced by their more substantive comments and their better performance on exams. The consistency of faculty evaluations from year to year and concurrent versus at the end of the course suggest that faculty tend not to make significant adjustments to student evaluations.