How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study

Introducing the Next Era in Assessment

In this introduction, the guest editors of the "Next Era in Assessment" special collection frame the invited papers by envisioning a next era in assessment of medical education, based on ideas developed during a summit that convened professional and educational leaders and scholars. The authors posit that the next era of assessment will focus unambiguously on serving patients and the health of society, reflect its sociocultural context, and support learners' longitudinal growth and development. As such, assessment will be characterized as transformational, development-oriented and socially accountable. The authors introduce the papers in this special collection, which represent elements of a roadmap towards the next era in assessment by exploring several foundational considerations that will make the next era successful. These include the equally important issues of (1) focusing on accountability, trust and power in assessment, (2) addressing implementation and contextualization of assessment systems, (3) optimizing the use of technology in assessment, (4) establishing infrastructure for data sharing and data storage, (5) developing a vocabulary around emerging sources of assessment data, and (6) reconceptualizing validity around patient care and learner equity. Attending to these priority areas will help leaders create authentic assessment systems that are responsive to learners' and society's needs, while reaping the full promise of competency-based medical education (CBME) as well as emerging data science and artificial intelligence technologies.

Critical Appraisal of Anesthesiology Educational Research for 2021

Background

Based on a review of anesthesiology education articles published in 2021, the authors conducted a critical appraisal to describe trends in the literature, highlight innovations in the field, and identify high-yield articles for clinician educators in anesthesiology.

Methods

After a database search (3 Ovid MEDLINE databases, Embase.com, ERIC [via FirstSearch], PsycINFO [via EBSCOhost], and PubMed), abstracts were screened by 2 independent reviewers based on inclusion criteria. Articles representing publications in both anesthesiology-specific journals and general medical education journals were included via manual search. Three randomly assigned raters reviewed and scored each quantitative article using a rubric. Two raters scored qualitative studies using a separate rubric designed for qualitative studies. Each article also received an overall quality rating used to create an additional list of recommended articles.

Results

The database search identified 1141 articles and an additional manual search identified 1497 articles. Of these, 67 articles met the inclusion criteria (61 quantitative, 6 qualitative). This article reports and summarizes the top 13 quantitative articles and top 2 qualitative papers.

Conclusions

This year we did not see as many articles describing curriculum to teach learners procedures, a topic heavily present in previous years. Also, analysis revealed an increase in articles focused on nontechnical skill education and a trend toward how to assess learner performance. Finally, 2 articles focused on gender issues in anesthesiology, an emerging area of interest.

Navigating the paradox: Exploring resident experiences of vulnerability

INTRODUCTION

Learning and growth in postgraduate medical education (PGME) often require vulnerability, defined as a state of openness to uncertainty, risk, and emotional exposure. However, vulnerability can threaten a resident's credibility and professional identity. Despite this tension, studies examining vulnerability in PGME are limited. As such, this study aims to explore residents' experiences of vulnerability, including the factors that influence vulnerability in PGME.

METHODS

Using a constructivist grounded theory approach, individual semi-structured interviews were conducted with 15 residents from 10 different specialities. Interview transcripts were coded and analysed iteratively. Themes were identified and relationships among themes were examined to develop a theory describing vulnerability in PGME.

RESULTS

Residents characterised vulnerability as a paradox represented by two overarching themes. 'Experiencing the tensions of vulnerability' explores the polarities between being a fallible, authentic learner and an infallible, competent professional. 'Navigating the vulnerability paradox' outlines the factors influencing the experience of vulnerability and its associated outcomes at the intrapersonal, interpersonal, and systems levels. Residents described needing to have the bandwidth to face the risks and emotional labour of vulnerability. Opportunities to build connections with social agents, including clinical teachers and peers, facilitated vulnerability. The sociocultural context shaped both the experience and outcomes of vulnerability as residents faced the symbolic mask of professionalism.

CONCLUSION

Residents experience vulnerability as a paradox shaped by intrapersonal, interpersonal, and systems level factors. These findings capture the nuance and complexity of vulnerability in PGME and offer insight into creating supportive learning environments that leverage the benefits of vulnerability while acknowledging its risks. There is a need to translate this understanding into systems-based change to create supportive PGME environments, which value and celebrate vulnerability.

Assessing supervisor versus trainee viewpoints of entrustment through cognitive and affective lenses: an artificial intelligence investigation of bias in feedback

The entrustment framework redirects assessment from considering only trainees' competence to decision-making about their readiness to perform clinical tasks independently. Since trainees and supervisors both contribute to entrustment decisions, we examined the cognitive and affective factors that underly their negotiation of trust, and whether trainee demographic characteristics may bias them. Using a document analysis approach, we adapted large language models (LLMs) to examine feedback dialogs (N = 24,187, each with an associated entrustment rating) between medical student trainees and their clinical supervisors. We compared how trainees and supervisors differentially documented feedback dialogs about similar tasks by identifying qualitative themes and quantitatively assessing their correlation with entrustment ratings. Supervisors' themes predominantly reflected skills related to patient presentations, while trainees' themes were broader-including clinical performance and personal qualities. To examine affect, we trained an LLM to measure feedback sentiment. On average, trainees used more negative language (5.3% lower probability of positive sentiment, p < 0.05) compared to supervisors, while documenting higher entrustment ratings (+ 0.08 on a 1-4 scale, p < 0.05). We also found biases tied to demographic characteristics: trainees' documentation reflected more positive sentiment in the case of male trainees (+ 1.3%, p < 0.05) and of trainees underrepresented in medicine (UIM) (+ 1.3%, p < 0.05). Entrustment ratings did not appear to reflect these biases, neither when documented by trainee nor supervisor. As such, bias appeared to influence the emotive language trainees used to document entrustment more than the degree of entrustment they experienced. Mitigating these biases is nonetheless important because they may affect trainees' assimilation into their roles and formation of trusting relationships.

History and physical exam: a retrospective analysis of a clinical opportunity

BACKGROUND

All learners at the Northern Ontario School of Medicine University complete a longitudinal integrated clerkship experience in their third year, which serves to improve learner experience with community and clinical acute and chronic health needs. Early in the program, Muskoka faculty (two of the 15 LIC sites of NOSM U) became aware that learners never had the occasion to complete a full history and physical exam on a real patient with complex needs. Recognizing this as a critical experience, a program was initiated to provide learners with this opportunity. This manuscript reports on the effectiveness and impact of this novel program and outlines the procedure developed to incorporate a similar program should communities see the relevance.

METHODS

Using a mixed method design, feedback was collected from learners and preceptors following the implementation of a novel learning opportunity for clinical clerks. Learners completed a full history and physical exam on volunteer complex patients, with supervision and immediate feedback. Using semi structured surveys, data was collected from each learner and preceptor to determine the program impact and optimize the program. Laurentian University research ethics board, certificate number 6021120.

RESULTS

Both learners and preceptors agreed this was a valuable experience for learners, a good use of their time and contributed to essential skills including, communication, time management and appropriate data collection. The use of real patients was reported to be very appropriate by learners and faculty and often highlighted gaps in the learner's knowledge that they were then able to address.

CONCLUSION

Feedback collected in this study confirms that providing medical learners the opportunity to complete a full history and physical exam with supervision and feedback was significantly beneficial from both a clinical and a skills-based aspect. Requiring learners to complete this task within the established period forced them to manage their time, focus on clinical consideration and remain on task. Enhancing learning opportunities is associated with improved outcomes and understanding in medical learners. Positive community experience is also related to learner retention, which is paramount for attracting new physicians in a time with significantly limited human health resources.

Evolution of the feedback conversation in anaesthesia education: a narrative review

Over the past century, education has been a core component for improving patient safety. The initial focus was developing a curriculum and an assessment process. In recent decades, the value of work-based learning has come to the fore. Learning from work, or experiential learning, requires reflection, which is critically dependent on external feedback. Conceptions of feedback have moved from a transactional information transfer from the supervisor to the trainee to a learner-centred and collaborative process occurring in a complex socio-cultural environment. In this narrative review we describe the evolution of the feedback conversation, provide a model synthesising the core concepts of feedback, and offer some guidance for the development of effective feedback in anaesthesia education.

It Takes a Village: Optimal Graduate Medical Education Requires a Deliberately Developmental Organization

Coaching is proposed as a means of improving the learning culture of medicine. By fostering trusting teacher-learner relationships, learners are encouraged to embrace feedback and make the most of failure. This paper posits that a cultural shift is necessary to fully harness the potential of coaching in graduate medical education. We introduce the deliberately developmental organization framework, a conceptual model focusing on three core dimensions: developmental communities, developmental aspirations, and developmental practices. These dimensions broaden the scope of coaching interactions. Implementing this organizational change within graduate medical education might be challenging, yet we argue that embracing deliberately developmental principles can embed coaching into everyday interactions and foster a culture in which discussing failure to maximize learning becomes acceptable. By applying the dimensions of developmental communities, aspirations, and practices, we present a six-principle roadmap towards transforming graduate medical education training programs into deliberately developmental organizations.

Examining trust between supervisors and trainees in the pediatric emergency department

Objective

Lack of trust between supervisors and trainees can have harmful consequences. Trust has been examined between physicians, patients, and staff in multiple settings but has not been examined in a bidirectional manner in the pediatric emergency department (PED). Debra Meyerson in 1996 discussed the theory of swift trust, a type of trust that develops quickly between temporary groups through reliance on categories. Meyerson describes categories as groups unified by social identities such as gender or profession. We explored the applicability of swift trust in the relationship between supervisors and trainees in the PED.

Methods

This multimethods study used qualitative interviews and the validated interpersonal mistrust trust measure (IMTM) to assess baseline trusting style. PED attendings from a single institution and residents from various training levels, specialties, and institutions rotating through the same PED were sampled until thematic saturation was reached. Interviews were analyzed using directed content analysis. IMTM scores were triangulated with interviewee's reported trusting styles.

Results

Seventeen PED attendings and 16 residents participated. Residents and attendings emphasized different factors that influence trust. Common factors across groups included affect, the need for vulnerability, and appropriate communication. Most described reliance on certain categories (level of training, specialty, etc.) when making trusting decisions. Categories such as gender and race were noted to be unimportant when making trusting decisions about others, although some participants believed these attributes played into decisions made by others. Quantitative data from the IMTM supported qualitative conclusions regarding trusting style.

Conclusions

The trusting relationship between supervisors and trainees in the PED dovetails with the swift trust theory. Common factors that influence trust reveal entry points for attendings and residents to improve trust and ultimately prevent negative patient outcomes. Future studies may examine how reliance on categories influences an individual's trust in their colleagues.

A phenomenology of direct observation in residency: Is Miller's 'does' level observable?

INTRODUCTION

Guidelines on direct observation (DO) present DO as an assessment of Miller's 'does' level, that is, the learner's ability to function independently in clinical situations. The literature, however, indicates that residents may behave 'inauthentically' when observed. To minimise this 'observer effect', learners are encouraged to 'do what they would normally do' so that they can receive feedback on their actual work behaviour. Recent phenomenological research on patients' experiences with DO challenges this approach; patients needed-and caused-some participation of the observing supervisor. Although guidelines advise supervisors to minimise their presence, we are poorly informed on how some deliberate supervisor participation affects residents' experience in DO situations. Therefore, we investigated what residents essentially experienced in DO situations.

METHODS

We performed an interpretive phenomenological interview study, including six general practice (GP) residents. We collected and analysed our data, using the four phenomenological lenses of lived body, lived space, lived time and lived relationship. We grouped our open codes by interpreting what they revealed about common structures of residents' pre-reflective experiences.

RESULTS

Residents experienced the observing supervisor not just as an observer or assessor. They also experienced them as both a senior colleague and as the patient's familiar GP, which led to many additional interactions. When residents tried to act as if the supervisor was not there, they could feel insecure and handicapped because the supervisor was there, changing the situation.

DISCUSSION

Our results indicate that the 'observer effect' is much more material than was previously understood. Consequently, observing residents' 'authentic' behaviour at Miller's 'does' level, as if the supervisor was not there, seems impossible and a misleading concept: misleading, because it may frustrate residents and cause supervisors to neglect patients' and residents' needs in DO situations. We suggest that one-way DO is better replaced by bi-directional DO in working-and-learning-together sessions.

Optimal training in communication model (OPTiCOM): A programmatic roadmap

OBJECTIVES

Teaching primary care residents patient communication skills is essential, complex, and impeded by barriers. We find no models guiding faculty how to train residents in the workplace that integrate necessary system components, the science of physician-patient communication training and competency-based medical education. The aim of this project is to create such a model.

METHODS

We created OPTiCOM using four steps: (1) communication educator interviews, analysis and theme development; (2) initial model construction; (3) model refinement using expert feedback; (4) structured literature review to validate, refine and finalize the model.

RESULTS

Our model contains ten interdependent building blocks organized into four developmental tiers. The Foundational value tier has one building block Naming relationship as a core value. The Expertize and resources tier includes four building blocks addressing: Curricular expertize, Curricular content, Leadership, and Time. The four building blocks in the Application and development tier are Observation form, Faculty development, Technology, and Formative assessment. The Language and culture tier identifies the final building block, Culture promoting continuous improvement in teaching communication.

CONCLUSIONS

OPTiCOM organizes ten interdependent systems building blocks to maximize and sustain resident learning of communication skills. Practice Implications Residency faculty can use OPTiCOM for self-assessment, program creation and revision.

Adaptability and learning Intraprofessional collaboration of residents during the COVID-19 pandemic

CONTEXT

The COVID-19 pandemic created a worldwide public health emergency, in which hospitals created new COVID departments and doctors from different disciplines had to work together. In the Netherlands, a large proportion of doctors in these departments were residents. With knowledge of the disease developing only gradually, the influx of COVID-19 patients called for adaptability, innovative work behavior, and intraprofessional collaboration (intraPC) between residents and between residents and medical specialists.

RESEARCH GOAL

This study investigates how the delivery of COVID-19 care in hospital settings altered the way residents develop their sense of adaptability and intraPC during their training.

METHODS

Sixteen semi-structured interviews were conducted with residents and medical specialists from various disciplines who worked at a COVID department or Intensive Care Unit (ICU) during the COVID pandemic in the Netherlands, focusing on adaptability and intraPC learning. Transcripts were analyzed using (thematic) template analysis.

RESULTS

Four themes that influenced learning during COVID care were identified: collective uncertainty, social cohesion and a sense of safety, the need for adaptive performance and intraPC learning. During the first wave, collective uncertainty about the unknown disease and the continuation of the crisis urged residents to adapt in order to take care of patients with a disease that was as yet unknown. The combination of collective uncertainty, social cohesion and a sense of safety, and the presence of different disciplines in one department promoted residents' intraPC learning. However, intraPC learning was not always the matter of course due to the scope of the crisis and the huge numbers of new patients.

CONCLUSION

Collective uncertainty affected the residents' adaptability. The combination of collective uncertainty, social cohesion, and the presence of different disciplines in one department promoted the residents' intraPC learning. An important facilitating factor for both adaptability and intraPC learning is a high level of social cohesion and safety. The physical and psychological proximity of supervisors is an important factor contributing to a safe learning environment. This study provides implications for practice for learning during postgraduate training in non-crisis settings.

Trust, power and learning in workplace-based assessment: The trainee perspective

For trainees to participate meaningfully in workplace-based assessment (WBA), they must have trust in their assessor. However, the trainee's dependent position complicates such trust. Understanding how power and trust influence WBAs may help us make them more effective learning opportunities. We conducted semi-structured interviews with 17 postgraduate anaesthesia trainees across Australia and New Zealand. Sensitised by notions of power, we used constructivist grounded theory methodology to examine trainees' experiences with trusting their supervisors in WBAs. In our trainee accounts, we found that supervisors held significant power to mediate access to learning opportunities and influence trainee progress in training. All episodes where supervisors could observe trainees, from simply working together to formal WBAs, were seen to generate assessment information with potential consequences. In response, trainees actively acquiesced to a deferential role, which helped them access desirable expertise and minimise the risk of reputational harm. Trainees granted trust based on how they anticipated a supervisor would use the power inherent in their role. Trainees learned to ration exposure of their authentic practice to supervisors in proportion to their trust in them. Trainees were more trusting and open to learning when supervisors used their power for the trainee's benefit and avoided WBAs with supervisors they perceived as less trustworthy. If assessment for learning is to flourish, then the trainee-supervisor power dynamic must evolve. Enhancing supervisor behaviour through reflection and professional development to better reward trainee trust would invite more trainee participation in assessment for learning. Modifying the assessment system design to nudge the power balance towards the trainee may also help. Modifications could include designated formative and summative assessments or empowering trainees to select which assessments count towards progress decisions. Attending to power and trust in WBA may stimulate progress towards the previously aspirational goal of assessment for learning in the workplace.