Trainees' perspectives of assessment messages: a narrative systematic review

Midwifery students and preceptors shared experiences of practice assessment: A qualitative descriptive study

BACKGROUND

Practice placement is a vital part of midwifery programs, enabling students to develop competence in caring for women and their babies under the supervision of preceptor midwives, who assess clinical skills while managing complex caseloads. Midwifery students and preceptor midwives are however contending with significant changes within the practice environment. These include changes to routes of entry to midwifery programmes, a move to a competency-based approach to assessment and increasing complexity of care required for women and their babies.

AIM

To examine the experiences of midwifery students and preceptor midwives with clinical assessment in an Irish university during a period of significant transformation in midwifery education and practice.

METHODS

A qualitative descriptive design was used to interview students (n = 32) and preceptors (n = 29). Data were analysed using content analysis.

RESULTS

Within an overarching theme of 'the context of assessment in practice', students and preceptors reported some commonalities in their experiences of practice assessment. A clear process of assessment was valued by participants, however, lack of continuity of preceptor and resource deficits, including inadequate staffing levels, affected this process. Preceptors and students highlighted the importance of support for practice assessment.

CONCLUSION

The findings emphasise the critical importance of the context of the practice environment in shaping stakeholders' experiences and is unique in eliciting the views of preceptors and students simultaneously. While resourcing midwifery services is essential, it must also encompass the resourcing of preceptorship, as safe practice depends on robust student assessment. Preceptors and students are supported in assessment by an explicit assessment process yet there are opportunities to further bolster supports, contributing to a quality practice learning environment for all. These insights hold significance for midwifery education and other practice-based professions.

Meaning making about performance: A comparison of two specialty feedback cultures

INTRODUCTION

Specialty trainees often struggle to understand how well they are performing, and feedback is commonly seen as a solution to this problem. However, medical education tends to approach feedback as acontextual rather than located in a specialty-specific cultural world. This study therefore compares how specialty trainees in surgery and intensive care medicine (ICM) make meaning about the quality of their performance and the role of feedback conversations in this process.

METHODS

We conducted a qualitative interview study in the constructivist grounded theory tradition. We interviewed 17 trainees from across Australia in 2020, eight from ICM and nine from surgery, and iterated between data collection and analytic discussions. We employed open, focused, axial and theoretical coding.

FINDINGS

There were significant divergences between specialties. Surgical trainees had more opportunity to work directly with supervisors, and there was a strong link between patient outcome and quality of care, with a focus on performance information about operative skills. ICM was a highly uncertain practice environment, where patient outcome could not be relied upon as a source of performance information; valued performance information was diffuse and included tacit emotional support. These different 'specialty feedback cultures' strongly influenced how trainees orchestrated opportunities for feedback, made meaning of their performance in their day-to-day patient care tasks and 'patched together' experiences and inputs into an evolving sense of overall progress.

DISCUSSION

We identified two types of meaning-making about performance: first, trainees' understanding of an immediate performance in a patient-care task and, second, a 'patched together' sense of overall progress from incomplete performance information. This study suggests approaches to feedback should attend to both, but also take account of the cultural worlds of specialty practice, with their attendant complexities. In particular, feedback conversations could better acknowledge the variable quality of performance information and specialty specific levels of uncertainty.

If we assess, will they learn? Students' perspectives on the complexities of assessment-for-learning

Introduction

Assessment can positively influence learning, however designing effective assessment-for-learning interventions has proved challenging. We implemented a mandatory assessment-for-learning system comprising a workplace-based assessment of non-medical expert competencies and a progress test in undergraduate medical education and evaluated its impact.

Methods

We conducted semi-structured interviews with year-3 and 4 medical students at McGill University to explore how the assessment system had influenced their learning in year 3. We conducted theory-informed thematic analysis of the data.

Results

Eleven students participated, revealing that the assessment influenced learning through several mechanisms. Some required little student engagement (i.e., feed-up, test-enhanced learning, looking things up after an exam). Others required substantial engagement (e.g., studying for tests, selecting raters for quality feedback, using feedback). Student engagement was moderated by the perceived credibility of the system and of the costs and benefits of engagement. Credibility was shaped by students' goals-in-context: becoming a good doctor, contributing to the healthcare team, succeeding in assessments.

Discussion

Our assessment system failed to engage students enough to leverage its full potential. We discuss the inherent flaws and external factors that hindered student engagement. Assessment designers should leverage easy-to-control mechanisms to support assessment-for-learning and anticipate significant collaborative work to modify learning cultures.

"Fit for the finals" - project report on a telemedical training with simulated patients, peers, and assessors for the licensing exam

Background

Undergraduate medical students take the licensing exam (M3) as a two-day oral-practical examination. The main requirements are to demonstrate history taking skills and coherent case presentations. The aim of this project was to establish a training in which students can test their communication skills during history taking and their clinical reasoning skills in focused case presentations.

Methods

In the newly developed training, final-year students took four telemedical histories in the role of physicians from simulated patients (SP). They received further findings for two SPs and presented these in a handover, in which they also received a handover of two SPs which they had not seen themselves. Each student presented one of the two received SPs in a case discussion with a senior physician. Feedback was given to the participants on their communication and interpersonal skills by the SPs with the ComCare questionnaire and on the case presentation by the senior physician. Sixty-two students from the universities of Hamburg and Freiburg in their final year participated in September 2022 and evaluated the training.

Results

Participants felt that the training was very appropriate for exam preparation. The SPs' feedback on communication and the senior physician's feedback on clinical reasoning skills received the highest ratings in importance to the students. Participants highly valued the practice opportunity for structured history taking and case presentation and would like to have more such opportunities in the curriculum.

Conclusion

Essential elements of the medical licensing exam can be represented, including feedback, in this telemedical training and it can be offered independent of location.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study

PURPOSE

In competency-based medical education, workplace-based assessment provides trainees with an opportunity for guidance and supervisors the opportunity to judge the trainees' clinical practice. Learning from assessment is enhanced when trainees reveal their thinking and are open to critique, which requires trust in the assessor. If supervisors knew more about how trainees come to trust them in workplace-based assessment, they could better engender trainee trust and improve trainees' learning experience.

METHOD

From August 2018 to September 2019, semistructured interviews were conducted with 17 postgraduate anesthesia trainees across Australia and New Zealand. The transcripts were analyzed using constructivist grounded theory methods sensitized by a sociocultural view of learning informed by Wenger's communities of practice theory.

RESULTS

Participants described a continuum from a necessary initial trust to an experience-informed dynamic trust. Trainees assumed initial trust in supervisors based on accreditation, reputation, and a perceived obligation of trustworthiness inherent in the supervisor's role. With experience and time, trainees' trust evolved based on supervisor actions. Deeper levels of trainee trust arose in response to perceived supervisor investment and allowed trainees to devote more emotional and cognitive resources to patient care and learning rather than impression management. Across the continuum from initial trust to experience-informed trust, trainees made rapid trust judgments that were not preceded by conscious deliberation; instead, they represented a learned "feel for the game."

CONCLUSIONS

While other factors are involved, our results indicate that the trainee behavior observed in workplace-based assessment is a product of supervisor invitation. Supervisor trustworthiness and investment in trainee development invite trainees to work and present in authentic ways in workplace-based assessment. This authentic engagement, where learners "show themselves" to supervisors and take risks, creates assessment for learning.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

Strategies for the assessment of competences during rheumatology training across Europe: results of a qualitative study

OBJECTIVES

To gain insight into current methods and practices for the assessment of competences during rheumatology training, and to explore the underlying priorities and rationales for competence assessment.

METHODS

We used a qualitative approach through online focus groups (FGs) of rheumatology trainers and trainees, separately. The study included five countries-Denmark, the Netherlands, Slovenia, Spain and the United Kingdom. A summary of current practices of assessment of competences was developed, modified and validated by the FGs based on an independent response to a questionnaire. A prioritising method (9 Diamond technique) was then used to identify and justify key assessment priorities.

RESULTS

Overall, 26 participants (12 trainers, 14 trainees) participated in nine online FGs (2 per country, Slovenia 1 joint), totalling 12 hours of online discussion. Strong nationally (the Netherlands, UK) or institutionally (Spain, Slovenia, Denmark) standardised approaches were described. Most groups identified providing frequent formative feedback to trainees for developmental purposes as the highest priority. Most discussions identified a need for improvement, particularly in developing streamlined approaches to portfolios that remain close to clinical practice, protecting time for quality observation and feedback, and adopting systematic approaches to incorporating teamwork and professionalism into assessment systems.

CONCLUSION

This paper presents a clearer picture of the current practice on the assessment of competences in rheumatology in five European countries and the underlying rationale of trainers' and trainees' priorities. This work will inform EULAR Points-to-Consider for the assessment of competences in rheumatology training across Europe.

Resident impression management within feedback conversations: A qualitative study

INTRODUCTION

Medical education is moving to conceptualise feedback as a bidirectional learning conversation. Within this conversation, learners experience a tension between assessment and feedback. That perceived tension affects learners' outward performances. In this study, we aimed to characterise residents' experiences with this tension and its effect on learner authenticity within feedback conversations.

METHODS

In this constructivist grounded theory study, the authors were informed by Goffman's theory of impression management. During data analysis, Dweck's theory of mindset was adopted. The authors conducted semi-structured interviews with 15 internal medicine residents. Data collection and analysis were conducted iteratively, using constant comparison to identify themes coinciding with impression management and mindset, ultimately developing a theoretical model to help explain residents' responses to tensions within feedback conversations.

RESULTS

Residents constantly felt 'scrutinized', and this affected their engagement in feedback conversations. They staged a performance within those conversations, linked to their underlying mindset: growth or fixed. Growth mindset was characterised by a focus on development as a physician and was associated with asking questions and seeking opportunities for growth. Fixed mindset was characterised by a focus on achieving a favourable evaluation and was associated with a hesitation to ask questions when faced with uncertainty and admit opportunities for growth, because they were concerned about impression management. Context influenced mindset and impression management. Residents adopted a fixed mindset and managed impressions when they perceived the permanence or consequences of evaluations within feedback. Residents adopted a growth mindset when they trusted the supervisor.

DISCUSSION

Residents assess the context of feedback conversations, altering the authenticity of their behaviours. Context, including the perceptions of formal assessment and relationships with supervisors, affected residents' mindset and impression management. Providing space for relationship-building and clarifying the purpose and structure of assessment may be helpful in supporting effective learning conversations in graduate medical education.

"There Is a Lot of Change Afoot": A Qualitative Study of Faculty Adaptation to Elimination of Tiered Grades With Increased Emphasis on Feedback in Core Clerkships

PURPOSE

Medical schools responding to challenges with fairness, equity, learning environments, and student wellness have reconsidered clerkship grades. This study used the Concerns-Based Adoption Model (CBAM) to explore how faculty responded to a change in the approach to assessment from focus on grades toward focus on feedback.

METHOD

This qualitative study used an inductive approach to analyze data from semistructured interviews with teaching faculty and education leaders at University of California, San Francisco, School of Medicine 6 months following the elimination of traditional tiered clerkship grades. Participants were recruited in 2019 using purposive sampling. Interview questions addressed participants' perceptions of the new approach to assessment and grading that emphasizes assessment for learning, including advantages and concerns. The authors analyzed data using thematic analysis informed by sensitizing concepts from CBAM.

RESULTS

Nineteen faculty participated (11 medicine, 8 surgery). Faculty expressed optimism about the impact of the change on their clinical educator roles and sought faculty development to enhance feedback and assessment skills. Perceived benefits to students addressed learning and wellness, with concern for students' motivation and professional development. Faculty shared uncertainty about their roles and larger systemic consequences involving the residency match and institutional reputation. Participants acknowledged that traditional grading is imperfect, yet felt a departure from this system represented significant culture change.

CONCLUSIONS

Faculty perceptions of the assessment and grade change in 2 large departments demonstrated tensions around grading. Their descriptions of change as an ongoing process aligned with CBAM and required them to consider new approaches and perspectives. While discourse about assessment and grading frequently focuses on learners, these findings support institutional consideration of the impact of changes in assessment on teaching faculty.

Use of work-based assessments in postgraduate medical education

Work-based assessments are ubiquitous in postgraduate medical training in the UK. This article discusses the variety of these assessments and explores barriers to their use and solutions for improving the educational value of these tools for adult learners. The focus should be on feedback and learning rather than assessment, and this may promote discussion of more challenging scenarios where the opportunity for learning is greater. Mobile devices may help reduce the administrative, geographical and time constraints of completing work-based assessments.