An Examination of Self-Reported Assessment Activities Documented by Specialist Physicians for Maintenance of Certification

Misperceptions and Missed Opportunities: A Qualitative Analysis of Barriers to Evaluating Surgical Teachers

OBJECTIVE

Teaching skills can be improved with written evaluations from learners. In addition to this primary purpose, teaching assessments are used in other aspects of faculty development including appointments, advancement and in some cases, compensation. Surgical trainees' willingness to provide meaningful assessments of their teachers is variable but the reasons for this are ill-defined. This study aims to elucidate surgical residents' perceptions regarding barriers to providing useful feedback to their teachers.

DESIGN

A qualitative, semi-structured confidential interview approach was used. A demographically diverse cohort of surgical residents in an urban university-based program was invited to participate. Interviews explored experiences and perceptions of teaching assessments. Specific attention was paid to understand perceptions of barriers; topics including utility, anonymity, time burden, and others were explored. Interviews were transcribed verbatim with identifiers removed from transcripts before analysis. All data was double coded to ensure accuracy with the development of a codebook until thematic exhaustion was reached.

SETTING

Yale New Haven Hospital is an academic, university-based medical center with approximately 70 residents in the general surgery program and approximately 170 surgical faculty.

PARTICIPANTS

A total of 21 residents completed individual or small group interviews. A theoretically driven sampling technique was used to recruit participants and maximize diversity. Individuals with varying backgrounds including PGY year, gender, age, IMG status, race, academic rank, research background and surgical division were asked to participate.

RESULTS

A total of 21 residents completed individual or small group interviews. Coding and analysis revealed 4 principal motifs: (1). Process- The process to complete assessment instruments is time-consuming and cumbersome to complete during the busy and acute surgical workday while failing to accurately address important aspects of surgical teaching. (2). Utility- Respondents reported uncertainty as to the downstream utility of the assessments, and a lack of confidence that the assessments would be used for faculty growth and improvement. (3). Resident Standing- Respondents described a lack of training, knowledge, skills, and empowerment to assess their teachers. (4). Perceived Consequences- Residents noted concern for identification, future autonomy, and other potential negative career consequences due to small resident sample sizes, recognizable experiences with attendings, and perceived power dynamics.

CONCLUSIONS

This study elucidates the perceptions of surgical trainees regarding barriers to providing feedback and assessments of their faculty. Although limited to a single-institution study, residents observed the current system does not allow for honest and accurate evaluations of surgical teachers. The extensive overlap between motifs highlights the need for a holistic approach to address these interconnected themes before teaching evaluations can be honest and productive. Importantly, it is also the first to identify residents' perceived lack of skill and sense of disempowerment to provide constructive faculty assessment. Due to the limited scope of the single-institution study, further verification and studies are needed to improve the quality of faculty feedback and assessment of surgical teachers.

Recasting Assessment in Continuing Professional Development as a Person-Focused Activity

ABSTRACT

In this article, we examine assessment as conceptualized and enacted in continuing professional development (CPD). Assessment is pervasive throughout the life of an individual health professional, serving many different purposes compounded by varied and unique contexts, each with their own drivers and consequences, usually casting the person as the object of assessment. Assessment is often assumed as an included part in CPD development conceptualization. Research on assessment in CPD is often focused on systems, utility, and quality instead of intentionally examining the link between assessment and the person. We present an alternative view of assessment in CPD as person-centered, practice-informed, situated and bound by capability, and enacted in social and material contexts. With this lens of assessment as an inherently personal experience, we introduce the concept of subjectification, as described by educationalist Gert Biesta. We propose that subjectification may be a fruitful way of examining assessment in a CPD context. Although the CPD community, researchers, and educators consider this further, we offer some early implications of adopting a subjectification lens on the design and enactment of assessment in CPD.

A Qualitative Study to Understand the Cultural Factors That Influence Clinical Data Use for Continuing Professional Development

INTRODUCTION

The use of data to inform lifelong learning has become increasingly important in continuing professional development (CPD) practice. Despite the potential benefits of data-driven learning, our understanding of how physicians engage in data-informed learning activities, particularly for CPD, remains unclear and warrants further study. The purpose of this study was to explore how physicians perceive cultural factors (individual, organizational, and systemic) that influence the use of clinical data to inform lifelong learning and self-initiated CPD activities.

METHODS

This qualitative study is part of an explanatory sequential mixed-methods study examining data-informed learning. Participants were psychiatrists and general surgeons from Canada and the United States. Recruitment occurred between April 2019 and November 2019, and the authors conducted semistructured telephone interviews between May 2019 and November 2019. The authors performed thematic analysis using an iterative, inductive method of constant comparative analysis.

RESULTS

The authors interviewed 28 physicians: 17 psychiatrists (61%) and 11 general surgeons (39%). Three major themes emerged from the continuous, iterative analysis of interview transcripts: (1) a strong relationship between data and trust, (2) a team-based approach to data-informed learning for practice improvement, and (3) a need for organizational support and advocacy to put data into practice.

CONCLUSION

Building trust, taking a team-based approach, and engaging multiple stakeholders, such as data specialists and organizational leadership, may significantly improve the use of data-informed learning. The results are situated in the existing literature, and opportunities for future research are summarized.

Only When They Seek: Exploring Supervisor and Resident Perspectives and Positions on Upward Feedback

INTRODUCTION

Verbal feedback from trainees to supervisors is rare in medical education, although valuable for improvement in teaching skills. Research has mostly examined narrative comments on resident evaluations of their supervisors. This study aimed to explore supervisors' and residents' beliefs and experiences with upward feedback, along with recommendations to initiate and facilitate effective conversations.

METHODS

Using 60-minute focus group discussions, a previous study explored opinions of internal medicine residents and clinical supervisors at the Brigham and Women's Hospital regarding the impact of institutional culture on feedback conversations. For this study, we conducted a thematic analysis of the transcribed, anonymous data to identify key concepts pertaining only to verbal upward feedback, through the theoretical lens of Positioning theory.

RESULTS

Twenty-two supervisors and 29 residents participated in three and five focus groups, respectively. Identified themes were mapped to three research questions regarding (1) existing beliefs (lack of impact, risks to giving supervisors feedback, need for preparation and reflection), (2) experiences (nonspecific language, avoidance of upward feedback, bypassing the supervisor), and (3) recommended approaches (setting clear expectations, seeking specific feedback, emphasizing interest in growth).

DISCUSSION

Study participants appeared to assume learner-teacher positions during feedback conversations, resulting in residents' concerns of adverse consequences, beliefs that supervisors will neither accept feedback nor change their behaviors, and avoidance of constructive upward feedback. Residents suggested that emphasis on mutual professional growth and regular feedback seeking by supervisors could encourage them to take on the role of feedback providers. Their recommendations could be a valuable starting point for faculty development initiatives on upward feedback.

Family Physician Quality Improvement Plans: A Realist Inquiry Into What Works, for Whom, Under What Circumstances

INTRODUCTION

Evaluation of quality improvement programs shows variable impact on physician performance often neglecting to examine how implementation varies across contexts and mechanisms that affect uptake. Realist evaluation enables the generation, refinement, and testing theories of change by unpacking what works for whom under what circumstances and why. This study used realist methods to explore relationships between outcomes, mechanisms (resources and reasoning), and context factors of a national multisource feedback (MSF) program.

METHODS

Linked data for 50 physicians were examined to determine relationships between action plan completion status (outcomes), MSF ratings, MSF comments and prescribing data (resource mechanisms), a report summarizing the conversation between a facilitator and physician (reasoning mechanism), and practice risk factors (context). Working backward from outcomes enabled exploration of similarities and differences in mechanisms and context.

RESULTS

The derived model showed that the completion status of plans was influenced by interaction of resource and reasoning mechanisms with context mediating the relationships. Two patterns were emerged. Physicians who implemented all their plans within six months received feedback with consistent messaging, reviewed data ahead of facilitation, coconstructed plan(s) with the facilitator, and had fewer risks to competence (dyscompetence). Physicians who were unable to implement any plans had data with fewer repeated messages and did not incorporate these into plans, had difficult plans, or needed to involve others and were physician-led, and were at higher risk for dyscompetence.

DISCUSSION

Evaluation of quality improvement initiatives should examine program outcomes taking into consideration the interplay of resources, reasoning, and risk factors for dyscompetence.

Rethinking Our Annual Congress-Meeting the Needs of Specialist Physicians by Partnering With Provincial Simulation Centers

Canada's maintenance of certification programs for physicians has evolved to emphasize assessment activities. Our organization recognized the importance of offering more practice assessment opportunities to our members to enhance their practice and help them comply with a regulation from our provincial professional body related to ongoing continuing education. This led us to rethink our annual congress and enrich the program with a curriculum of interdisciplinary simulation sessions tailored to meet the needs of a broad audience of specialists. Our challenges are similar to those of many national specialty societies having limited access to simulation facilities, instructors, and simulation teams that can cover the breadth and scope of perceived and unperceived simulation needs for their specialty. Our innovative solution was to partner with local experts to develop 22 simulation sessions over the past three years. The response was very positive, drawing 867 participants. Over 95% of participants either agreed or strongly agreed that their simulation session (1) met their learning objectives, (2) was relevant for their practice, and (3) encouraged them to modify their practice. Narrative comments from a survey sent to the 2018 participants four months after their activity indicated several self-reported changes in their practice or patient outcomes. We were able to centralize offers from organizations that had previously worked in silo to develop simulation sessions meeting the needs of our members. Proposing simulation sessions allowed our organization to establish long-term partnerships and to expend our "educational toolbox" to address skill gaps not usually addressed during annual meetings.