Is it me or is it them? Factors that influence the passing of underperforming students

"They were very very nice but just not very good": The interplay between resident-supervisor relationships and assessment in the emergency setting

Purpose

Clinical supervisors hesitate to report learner weaknesses, a widely documented phenomenon referred to as "failure to fail." They also struggle to discuss weaknesses with learners themselves. Their reluctance to report and discuss learner weaknesses threatens the validity of assessment-of-learning decisions and the effectiveness of assessment for learning. Personal and interpersonal factors have been found to act as barriers to reporting learners' difficulties, but the precise role of the resident-supervisor relationship remains underexplored, specifically in the emergency setting. This study aims to better understand if and how factors related to the resident-supervisor relationship are involved in assessment of and for learning in the emergency setting.

Methods

We conducted a qualitative study, using semistructured interviews of 15 clinical supervisors in emergency medicine departments affiliated with our institution. Transcripts were independently coded by three members of the team using an iterative mixed deductive-inductive thematic analysis approach. The team then synthesized the coding and discussed analysis following guidelines for thematic analysis.

Results

Participating emergency medicine supervisors valued resident-supervisor relationships built on collaboration and trust and believed that such relationships support learning. They described how these relationships influenced assessment of and for learning and how in turn assessment influenced the relationship. Almost all profiles of resident-supervisor relationships in our study could hinder the disclosing of resident weaknesses, through a variety of mechanisms. To protect residents and themselves from the discomfort of disclosing weaknesses and to avoid deteriorating the resident-supervisor relationship, many downplayed or even masked residents' difficulties. Supervisors who described themselves as able to provide negative assessment of and for learning often adopted a more distant or professional stance.

Conclusions

This study contributes to a growing literature on failure to fail by confirming the critical impact that the resident-supervisor relationship has on the willingness and ability of emergency medicine supervisors to play their part as assessors.

Assessor discomfort and failure to fail in clinical performance assessments

BACKGROUND

Assessment of trainee performance in the workplace is critical to ensuring high standards of clinical care. However, some supervisors find the task to be challenging, and may feel unable to deliver their true judgement on a trainee's performance. They may 'keep MUM' (that is, keep mum about undesirable messages) and fail to fail an underperforming trainee. In this study, we explore the effect of discomfort on assessors.

METHODS

Using a survey method, supervisors of trainees in the Australasian College of Dermatologists were asked to self-report experiences of discomfort in various aspects of trainee workplace assessment and for their engagement in MUM behaviours including failure to fail.

RESULTS

Sixty-one responses were received from 135 eligible assessors. 12.5% of assessors self-reported they had failed to fail a trainee and 18% admitted they had grade inflated a trainee's score on a clinical performance assessment in the previous 12-month period. Assessors who reported higher levels of discomfort in the clinical performance assessment context were significantly more likely to report previously failing to fail a trainee. The study did not reveal significant associations with assessor demographics and self-reports of discomfort or MUM behaviours.

CONCLUSIONS

This study reveals the impact of assessor discomfort on the accuracy of assessment information and feedback to trainees, including as a contributing factor to the failure to fail phenomenon. Addressing assessor experience of discomfort offers one opportunity to impact on the complex and multifactorial issue that failure to fail represents.

Longitudinal assessment of undergraduate dental students: Building evidence for validity

PURPOSE

To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations.

METHODS

Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha.

RESULTS

Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included.

CONCLUSIONS

There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.

'But what if you miss something …?': factors that influence medical student consideration of cost in decision making

CONTEXT

Cost-conscious care is critical for healthcare sustainability but evidence suggests that most doctors do not consider cost in their clinical decision making. A critical step in changing this is understanding the barriers to encouraging behaviours and attitudes related to cost-conscious care. We therefore conducted a qualitative study to address the research question: what factors influence consideration of cost in emergency medicine (ED) clinical decision making?

METHODS

This was a qualitative focus group study using patient vignettes to explore attitudes towards cost-conscious clinical decision making. Participants were Year 4 and Year 5 medical students from Singapore, a country with a fee-for-service healthcare system. After a data-driven initial data analysis, and to make sense of a multitude of factors impacting on cost conscious care, we selected Fishbein's integrative model of behavioural prediction to underpin secondary data analysis.

RESULTS

Via four focus groups with 21 participants, we identified five main themes relevant to the integrative model of behavioural prediction. These were: attitudes towards considering cost when managing a patient (e.g., "better safe than sorry"); normative beliefs (e.g., doing what others do, perceptions of patient wishes); efficacy beliefs (e.g., no authority to take decisions or challenge); skills and knowledge (e.g., little knowledge of costs), and environmental constraints (e.g., the nature of the healthcare system).

DISCUSSION

Medical students do not consider cost in their clinical decision making due to numerous factors, of which lack of knowledge of costs is but one. While some of the factors identified reflect those found in previous studies with residents and fully-trained staff, and in other contexts, theory driven analysis added value in that it facilitated a richer exploration of why students do not consider cost in clinical decision making. Our findings provide insight to inform how best to engage and empower educators and learners in teaching and learning about cost-conscious care.

Failing to fail phenomena

INTRODUCTION

Clinical competence is the backbone of competence-based dental education. Over time, there has been a paradigm shift toward training students who are capable of independent practice, as opposed to mere academic success.

METHODS

A mixed-method study was undertaken by anonymised email questionnaire to all restorative tutors at the UK Dental School. Demographics and teaching experience were ascertained, along with key questions on the utilisation of online assessment software iDentity. The assessment process for tutors was explored, and barriers experienced when grading students were reported.

RESULTS

The questionnaire was sent to all 51 restorative tutors with a response rate of 59% (n = 30). Only 3.5% of tutors provided verbal feedback and grading to students in person, with 20.7% only completing iDentity gradings following an email reminder. The majority of staff (93.3%) felt comfortable in raising concerns; however, one of the three clinical tutors admitted they had allowed a failing student to a pass. Qualitative analysis demonstrated several themes why tutors were reluctant to fail students: maintaining good relationships, limited supervision, time delay of grading, one-off event and the student's first attempt.

CONCLUSIONS

Grading students as competent as a one-off experience could potentially mask a recurring problem with a student, in turn impacting the student's ability to assess their own weakness and believe themselves to be competent, and potentially be overconfident. Fair and accurate assessment has a significant benefit to student and staff, enabling targeted development to motivate the students and improve the quality of care provided to the patients.

Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training

Background

Remediation of struggling learners in pulmonary and critical care fellowship programs is a challenge, even for experienced medical educators.

Objective

This evidence-based narrative review provides a framework program leaders may use to address fellows having difficulty achieving competency during fellowship training.

Methods

The relevant evidence for approaches on the basis of each learner’s needs is reviewed and interpreted in the context of fellowship training in pulmonary medicine and critical care. Issues addressed include bias in fellow assessments and remediation, the impacts of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the specific challenges of pulmonary and critical care fellowship programs, a brief review of relevant legal issues, guidance on building and leveraging program resources, and a discussion of learner outcomes.

Results

This results in a concise, evidence-based toolkit for program leaders based around four pillars: early identification, fellow assessment, collaborative intervention, and reassessment. Important concepts also include the need for documentation, clear and written communication, and fellow-directed approaches to the creation of achievable goals.

Conclusion

Evidence-based remediation helps struggling learners in pulmonary and critical care fellowship to improve their ability to meet Accreditation Council for Graduate Medical Education (ACGME) milestones.

Evaluating Clinical Educators' Competence in an East Asian Context: Who Values What?

Background

How to evaluate clinical educators is an important question in faculty development. The issue of who are best placed to evaluate their performance is also critical. However, the whos and the hows of clinical educator evaluation may differ culturally. This study aims to understand what comprises suitable evaluation criteria, alongside who is best placed to undertake the evaluation of clinical educators in medicine within an East Asian culture: specifically Taiwan.

Methods

An 84-item web-based questionnaire was created based on a literature review and medical educational experts' opinions focusing on potential raters (i.e., who) and domains (i.e., what) for evaluating clinical educators. Using purposive sampling, we sent 500 questionnaires to clinical educators, residents, Post-Graduate Year Trainees (PGYs), Year-4~6/Year-7 medical students (M4~6/M7) and nurses.

Results

We received 258 respondents with 52% response rate. All groups, except nurses, chose “teaching ability” as the most important domain. This contrasts with research from Western contexts that highlights role modeling, leadership and enthusiasm. The clinical educators and nurses have the same choices of the top five items in the “personal qualities” domain, but different choices in “assessment ability” and “curriculum planning” domains. The best fit rater groups for evaluating clinical educators were educators themselves and PGYs.

Conclusions

There may well be specific suitable domains and populations for evaluating clinical educators' competence in East Asian culture contexts. Further research in these contexts is required to examine the reach of these findings.

The shift from disbelieving underperformance to recognising failure: A tipping point model

CONTEXT

Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed.

METHODS

Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process.

RESULTS

The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm.

CONCLUSION

Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

Facing the "Fear of Failure": Veterinary Students in Clinical Rotations

Failing a student is difficult for both educator and student, but administering a failing grade is critical for protecting and ensuring adequate learning for an unsafe student. The failure to fail clinical students has been commonly reported and explored among educators in the human health professions but has not been formally addressed in veterinary education. Forty-three participants attending the Veterinary Educators Collaborative symposium were surveyed concerning their attitudes and experiences failing clinical veterinary students. Results indicated that the failure to fail phenomenon exists among veterinary educators, as the majority of veterinary educators often felt reluctant and unprepared to fail a student on clinical rotations. The most common barriers to failing students were institutional culture and unsatisfactory assessor development or evaluation tools. Veterinary educators must face this fear of failure and explore strategies to overcome existing barriers that can ultimately transform student failure into success.

Failure to fail - Factors affecting faculty decisions to pass underperforming nursing students in the clinical setting: A quantitative study

AIM

The purpose of the study was to explore the relationship between face-implicating factors and faculty's likelihood of failing students in the clinical setting who do not meet passing criteria.

BACKGROUND

Clinical nursing faculty members struggle to assign failing grades to underperforming students in the clinical setting; this is known as failure to fail. Qualitative literature has revealed common factors for failure to fail; however, quantitative studies are required to determine the extent to which those factors affect faculty's decision-making process.

DESIGN

A quantitative, descriptive design was used.

METHODS

Snowball sampling was used to recruit participants from CCNE- and ACEN-accredited nursing programs to complete an online survey. There were 353 responses to the survey (a 30% return rate) and 327 usable responses. Eligibility criteria included pre-licensure nursing faculty members who had taught in the clinical setting within the past three years. The tool used for the study was adapted from Dibble's (2014) tool, which explored face-implicating factors' impact on the transmission of bad news.

RESULTS

Respondents who did not commit failure to fail (F2FN) disagreed more strongly with every survey item than those who committed failure to fail (F2FY). The differences in mean scores were compared and 64% of those differences were statistically significant (p < 0.05). Respondents who did not commit failure to fail were less affected by the face-implicating factors than those who committed failure to fail.

CONCLUSIONS

the null hypothesis was rejected; a direct connection was found between face-implicating factors and faculty's likelihood of passing students in the clinical setting who do not meet passing criteria.

The recommended description of an entrustable professional activity: AMEE Guide No. 140

Entrustable professional activities (EPAs) have received much attention in the literature since they were first proposed in 2005. Useful guidelines, workshops, courses, and conferences have supported faculty in developing programs and designing assessment procedures using EPAs and entrustment decision-making. Yet, the need for clarification remains, particularly as more programs make the step from design to implementation.Well-written EPAs provide a natural construct to establish the outcome of training. To be useful, EPAs require more than a suitable title. This AMEE Guide elaborates eight sections of a full EPA description, and provides explanations and justifications for each. These sections are: title; specification and limitations; risks in case of failure; most relevant competency domains; knowledge, skills, attitudes and experiences; information sources to assess progress and support summative entrustment; entrustment/supervision level expected at which stage of training; and time period to expiration if not practiced.Constructing fully elaborated EPAs creates a shared mental model amongst learners and programs, informs competency-based curriculum design, directs ad-hoc and formal entrustment decision-making, and provides standards for certifying bodies and boundaries for scope of practice. The framework intends to support curricular leaders looking to adopt new EPAs, or revise and define established EPAs for competency-based education.

Management of residents in difficulty in a Swiss general internal medicine outpatient clinic: Change is necessary!

AIMS OF THE STUDY

Residents in difficulty are a major cause for concern in medical education, with a prevalence of 7-15%. They are often detected late in their training and cannot make use of remediation plans. Nowadays, most training hospitals in Switzerland do not have a specific program to identify and manage residents in difficulty. The aim of the study was to explore the challenges perceived by physicians regarding the process of identifying, diagnosing, and supporting residents in difficulty in a structured and programmatic way. We explored perceptions of physicians at different hierarchical levels (residents (R), Chief residents (CR), attending physicians (A), Chief Physician (CP)) in order to better understand these challenges.

METHODS

We conducted an exploratory qualitative study between December 2015 and July 2016. We asked volunteers from the Primary Care Division of the Geneva University Hospitals to partake to three focus groups (with CR, A, R) and one interview with the division's CP. We transcribed, coded, and qualitatively analyzed the three focus groups and the interview, using a content thematic approach and Fishbein's conceptual framework.

RESULTS

We identified similarities and differences in the challenges of the management of residents in difficulty on a programmatic way amongst physicians of different hierarchical levels. Our main findings: Supervisors (CR, A, CP) have good identification skills of residents in difficulty, but they did not put in place systematic remediation strategies.Supervisors (CR, A) were concerned about managing residents in difficulty. They were aware of the possible adverse effects on patient care, but "feared to harm" resident's career by documenting a poor institutional assessment.Residents "feared to share" their own difficulties with their supervisors. They thought that it would impact their career negatively.The four physician's hierarchical level reported environmental constraints (lack of funding, time constraint, lack of time and resources…).

CONCLUSION

Our results add two perspectives to specialized recommendations regarding the implementation of remediation programs for residents in difficulty. The first revolves around the need to identify and fully understand not only the beliefs but also the implicit norms and the feeling of self-efficacy that are shared by teachers and that are likely to motivate them to engage in the management of residents in difficulty. The second emphasizes the importance of analyzing these elements that constitute the context for a change and of identifying, in close contact with the heads of the institutions, which factors may favor or hinder it. This research action process has fostered awareness and discussions at different levels. Since then, various actions and processes have been put in place at the Faculty of Medicine in Geneva.

Redefining scholarship for health professions education: AMEE Guide No. 142

Scholarship is an integral aspect of academia. It shapes the practice of individuals and the field and is often used to inform career progression decisions and policies. This makes it high stakes. Yet it is complex, and ambiguous. Definitions vary and the term does not necessarily translate well across contexts. In this AMEE guide, our aim is to establish a contemporary definition of scholarship which is appropriate to health professions education (HPE). Our specific objectives are to provide guidance to support colleagues in their career development as professional educators and to challenge biases and assumptions about scholarship which may still exist in educational systems and structures. Ultimately, we hope that this work will advance the stature/standing of scholarship in the field. We provide a general definition of scholarship and how this relates to the scholarship of teaching (and learning) (SoT[L]) and scholarly teaching. Drawing on Boyer's seminal work, we describe different types of scholarship and reflect on how these apply to HPE, before moving on to describe different types of engagement with scholarship in HPE, including scope of contribution and influence. Using cases and examples, we illustrate differences in scholarly engagement across stages of a career, contexts, and ways of engaging. We provide guidance on how to assess 'quality' of scholarship. We offer practical advice for health professions' educators seeking academic advancement. We advocate that institutional leaders consider their systems and structures, so that these align with faculty work patterns, and judge teaching and professional practice appropriately. We conclude by offering a new definition of scholarship in HPE.

Investigating faculty assessment of anesthesia trainees and the failing-to-fail phenomenon: a randomized controlled trial

PURPOSE

A growing body of evidence has shown that supervisors may "fail to fail" trainees even when they have judged their performance unsatisfactory. This has significant implications for the implementation of a nationwide competency-based education model of residency training. The objective of this study was to determine the incidence of "failing to fail" clearly underperforming residents.

METHODS

Study participants were recruited via an email invitation sent to all departments of anesthesia at each of the hospitals affiliated with the University of Toronto. They were randomized into a high-stakes (assessment would affect the resident's progress) or low-stakes (assessment would not affect the resident's progress) group and asked to assess the performance (fail or pass grade) of a struggling resident. Participants assessed a video depicting an actor managing a scripted simulation scenario. It contained several critical clinical mistakes constituting a clear fail performance. The purpose of the study was only disclosed following the assessment.

RESULTS

Of the 288 email invitations sent (144 in each group), 158 (54%) participants completed the study, with 93 in the high-stakes group and 65 in the low-stakes group. Twenty-eight participants (17.7%) failed to issue a failing grade, including 23.1% (15/65) in the high-stakes group and 13.9% (13/93) in the low-stakes group (P = 0.14).

CONCLUSIONS

Though often discussed, this is the first study to quantitatively show that the "failing-to-fail" phenomenon likely occurs during residency training performance evaluations. Passing underperforming learners can potentially affect patient safety and result in severe personal consequences to the learner. The results indicate the need for better performance assessment training for faculty members.

Seeing but not believing: Insights into the intractability of failure to fail

CONTEXT

Inadequate documentation of observed trainee incompetence persists despite research-informed solutions targeting this failure to fail phenomenon. Documentation could be impeded if assessment language is misaligned with how supervisors conceptualise incompetence. Because frameworks tend to itemise competence as well as being vague about incompetence, assessment design may be improved by better understanding and describing of how supervisors experience being confronted with a potentially incompetent trainee.

METHODS

Following constructivist grounded theory methodology, analysis using a constant comparison approach was iterative and informed data collection. We interviewed 22 physicians about their experiences supervising trainees who demonstrate incompetence; we quickly found that they bristled at the term 'incompetence,' so we began to use 'underperformance' in its place.

RESULTS

Physicians began with a belief and an expectation: all trainees should be capable of learning and progressing by applying what they learn to subsequent clinical experiences. Underperformance was therefore unexpected and evoked disbelief in supervisors, who sought alternate explanations for the surprising evidence. Supervisors conceptualised underperformance as: an inability to engage with learning due to illness, a life event or learning disorders, so that progression was stalled, or an unwillingness to engage with learning due to lack of interest, insight or humility.

CONCLUSION

Physicians conceptualise underperformance as problematic progression due to insufficient engagement with learning that is unresponsive to intensified supervision. Although failure to fail tends to be framed as a reluctance to document underperformance, the prior phase of disbelief prevents confident documentation of performance and delays identification of underperformance. The findings offer further insight and possible new solutions to address under-documentation of underperformance.

The role of previously undocumented data in the assessment of medical trainees in clinical competency committees

INTRODUCTION

The clinical competency committee (CCC) comprises a group of clinical faculty tasked with assessing a medical trainee's progress from multiple data sources. The use of previously undocumented data, or PUD, during CCC deliberations remains controversial. This study explored the use of previously undocumented data in conjunction with documented data in creating a meaningful assessment in a CCC.

METHODS

An instrumental case study of a CCC that uses previously undocumented data was conducted. A single CCC meeting was observed, followed by semi-structured individual interviews with all CCC members (n = 7). Meeting and interview transcripts were analyzed iteratively.

RESULTS

Documented data were perceived as limited by inaccurate or superficial data, but sometimes served as a starting point for invoking previously undocumented data. Previously undocumented data were introduced as summary impressions, contextualizing factors, personal anecdotes and, rarely, hearsay. The purpose was to raise a potential issue for discussion, enhance and elaborate an impression, or counter an impression. Various mechanisms allowed for the responsible use of previously undocumented data: embedding these data within a structured format; sharing relevant information without commenting beyond one's scope of experience; clarifying allowable disclosure of personal contextual factors with the trainee pre-meeting; excluding previously undocumented data not widely agreed upon in decision-making; and expecting these data to have been provided as direct feedback to trainees pre-meeting.

DISCUSSION

Previously undocumented data appear to play a vital part of the group conversation in a CCC to create meaningful, developmentally focused trainee assessments that cannot be achieved by documented data alone. Consideration should be given to ensuring the thoughtful incorporation of previously undocumented data as an essential part of the CCC assessment process.

Learner handover: Perspectives and recommendations from the front-line

INTRODUCTION

Current medical education models increasingly rely on longitudinal assessments to document learner progress over time. This longitudinal focus has re-kindled discussion regarding learner handover-where assessments are shared across supervisors, rotations, and educational phases, to support learner growth and ease transitions. The authors explored the opinions of, experiences with, and recommendations for successful implementation of learner handover among clinical supervisors.

METHODS

Clinical supervisors from five postgraduate medical education programs at one institution completed an online questionnaire exploring their views regarding learner handover, specifically: potential benefits, risks, and suggestions for implementation. Survey items included open-ended and numerical responses. The authors used an inductive content analysis approach to analyze the open-ended questionnaire responses, and descriptive and correlational analyses for numerical data.

RESULTS

Seventy-two participants completed the questionnaire. Their perspectives varied widely. Suggested benefits of learner handover included tailored learning, improved assessments, and enhanced patient safety. The main reported risk was the potential for learner handover to bias supervisors' perceptions of learners, thereby affecting the validity of future assessments and influencing the learner's educational opportunities and well-being. Participants' suggestions for implementation focused on who should be involved, when and for whom it should occur, and the content that should be shared.

DISCUSSION

The diverse opinions of, and recommendations for, learner handover highlight the necessity for handover to maximize learning potential while minimizing potential harms. Supervisors' suggestions for handover implementation reveal tensions between assessment-of and for-learning.

Conflict between clinician teachers and their students: the clinician perspective

The relationship between clinician teachers and their students is of major importance in medical education. However, there is little known about the effects on clinicians when conflict occurs with their students. What do clinicians perceive to be major causes of these conflicts? How do they react when and after conflict occurs? A phenomenological inquiry exploring the lived experience of 12 clinician teachers in medical schools was performed. The clinicians were selected using purposeful sampling and snowballing techniques. The interviews revolved around discussions based on episodes of conflict with medical students that the clinicians considered significant. The analysis and emergent themes were partially constructed around and informed by theories of conflict, and conflict management. A number of themes emerged from this study. Clinicians experienced that significant psychological and behavioural problems of students had a dominant impact on the likelihood and severity of conflict; these conflicts had a significant emotional impact on clinicians; though the responses to conflict varied, "avoidance" was a mechanism commonly used by clinicians and thus the assessment of attitudinal and behavioural professional issues in the workplace was problematic. This study shows how the clinician perspective to challenging student/clinician encounters impacts on the quality of education they are able to provide. We recommend medical schools consider these issues when designing their programs in order to develop and maintain clinician-teacher engagement and participation.

Guidelines: The dos, don'ts and don't knows of remediation in medical education

INTRODUCTION

Two developing forces have achieved prominence in medical education: the advent of competency-based assessments and a growing commitment to expand access to medicine for a broader range of learners with a wider array of preparation. Remediation is intended to support all learners to achieve sufficient competence. Therefore, it is timely to provide practical guidelines for remediation in medical education that clarify best practices, practices to avoid, and areas requiring further research, in order to guide work with both individual struggling learners and development of training program policies.

METHODS

Collectively, we generated an initial list of Do's, Don'ts, and Don't Knows for remediation in medical education, which was then iteratively refined through discussions and additional evidence-gathering. The final guidelines were then graded for the strength of the evidence by consensus.

RESULTS

We present 26 guidelines: two groupings of Do's (systems-level interventions and recommendations for individual learners), along with short lists of Don'ts and Don't Knows, and our interpretation of the strength of current evidence for each guideline.

CONCLUSIONS

Remediation is a high-stakes, highly complex process involving learners, faculty, systems, and societal factors. Our synthesis resulted in a list of guidelines that summarize the current state of educational theory and empirical evidence that can improve remediation processes at individual and institutional levels. Important unanswered questions remain; ongoing research can further improve remediation practices to ensure the appropriate support for learners, institutions, and society.

Struggling with strugglers: using data from selection tools for early identification of medical students at risk of failure

BACKGROUND

Struggling medical students is an under-researched in medical education. It is known, however, that early identification is important for effective remediation. The aim of the study was to determine the predictive effect of medical school admission tools regarding whether a student will struggle academically.

METHODS

Data comprise 700 students from the University of New South Wales undergraduate medical program. The main outcome of interest was whether these students struggled during this 6-year program; they were classified to be struggling they failed any end-of-phase examination but still graduated from the program. Discriminate Function Analysis (DFA) assessed whether their pre-admission academic achievement, Undergraduate Medicine Admission Test (UMAT) and interview scores had predictive effect regarding likelihood to struggle.

RESULTS

A lower pre-admission academic achievement in the form of Australian Tertiary Admission Rank (ATAR) or Grade Point Average (GPA) were found to be the best positive predictors of whether a student was likely to struggle. Lower UMAT and poorer interview scores were found to have a comparatively much smaller predictive effect.

CONCLUSION

Although medical admission tests are widely used, medical school rarely use these data for educational purposes. The results of this study suggest admission test data can predict who among the admitted students is likely to struggle in the program. Educationally, this information is invaluable. These results indicate that pre-admission academic achievement can be used to predict which students are likely to struggle in an Australian undergraduate medicine program. Further research into predicting other types of struggling students as well as remediation methods are necessary.

Educator as Diagnostician, Judge and Confidant: a positioning analysis of medical student support encounters

Medical schools, programs and educators are increasingly expected to address medical student stress and wellbeing, yet also ensure student competence and fitness to practice. Educators play a central role in supporting students when evaluating a student's concerns and in deciding whether support and/or sanction should be offered. It is not known how educators approach or resolve such potentially contradictory responses. We conducted an interview study of 21 medical educators from a range of  professional backgrounds across 11 on-campus and clinical teaching sites. Using Positioning Theory to inform our thematic analysis, we found that participants adopted an overarching position of Diagnostician, and at times, two alternative positions, the Judge and the Confidant when supporting students. In their narratives of support encounters, individual students were positioned as Good Students or Troubling Students. For most, educator positions were fluid and responsive to the storylines enacted in encounters. Rigidly adopting Judge or Confidant positions could lead to "failure to fail" and violations of professional boundaries. Positioning Theory locates student support in a moral space and helps explain the consternation experienced by educators when support is not effective. Positioning analysis offers a language, and metaphors which are meaningful to educators, for framing discussion and reviews of support practices and progression decisions. Such insights could encourage reflective practice and guide further research to inform practice when students with troubling concerns and persistently borderline performances require support.

Twelve tips for clinicians dealing with uncertainty when assessing learners

Clinician educators often experience distress caused by uncertainty regarding how effectively to participate in assessment practices in a way that supports both their programs and their students. Uncertainty is a common state for clinicians, particularly for those who see patients with early or ill-defined illness presentations. While clinicians often feel ill at ease when facing uncertainty in the clinical realm, becoming comfortable with uncertainty and learning to manage such states are now recognized as vital components of clinical practice. Clinicians, as a result, have adopted a series of strategies to lessen the unease that uncertainty can create. While similar experiences plague clinician educators placed in assessment roles, much less attention has been given to how we can support individuals in the education setting. Here, the distress of uncertainty may be greater due to clinician educators having less experience with assessment practices. Fortunately, strategies that are effective in the clinical domain can be translated into the assessment realm to accommodate uncertainty when assessing learners. In this 12 tips article we offer guidance on the translation of such strategies.

Failing to Fail nursing students among mentors: A confirmatory factor analysis of the Failing to Fail scale

AIM

The aim was to explore the psychometric properties with respect to the internal consistency reliability of the subject-specific questionnaire "Failing to Fail."

DESIGN

Cross-sectional study.

METHODS

Exploratory factor analysis with varimax rotation. A confirmatory factor analysis was used to examine the factor structure of the "Failing to Fail" scale. The sample included 336 Norwegian nurse mentors.

RESULTS

The confirmatory factor analysis confirmed a five-factor structure of the "Failing to Fail" scale with adequate model fit. The factors were named as: (a) Insufficient mentoring competence; (b) Insufficient support in the working environment; (c) Emotional process dominates the assessment; (d) Insufficient support from the university; and (e) Decision-making detached from learning outcomes. The scale proved to be feasible to test whether mentors are Failing to Fail nursing students. The confirmatory factor analysis model supported the predictive validity of the "Failing to Fail" scale.

Failure of faculty to fail failing medical students: Fiction or an actual erosion of professional standards?

OBJECTIVES

Literature has shown that some assessors assign passing grades to medical students who, in fact, should not have passed. This inability of the faculty to fail underperforming students can jeopardise the reputation of professional programs, be it in the medical field or beyond. Simultaneously, weak students become incompetent physicians and, thus, endanger the community they serve. The impetus for conducting this systematic review was to identify barriers to faculty in failing struggling medical students.

METHODS

The databases of MEDLINE, Scopus, Wiley online library, Cochrane library, OVID, Taylor and Francis, CINAHL, Springer link, ProQuest, and ISI Web of knowledge were searched using Medical Subject Headings (MeSH) terms 'Faculty failure' AND 'Failing students' AND 'Failure to fail' OR 'Assessment'. The data were synthesised, and the results were analysed.

RESULTS

This search showed a wealth of barriers to faculty contributing to a 'failure to fail' such as their concerns about legal action and an appeals process; the stress of failing students; a lack of knowledge about proper documentation; unavailability of support, resources, and offices for faculty; absence of administrative guidelines; and complex dismissal procedures discouraging the faculty from failing students.

CONCLUSION

Institutional faculty development programs and training workshops should facilitate the education of supervisors and assessors for timely evaluation and regular documentation of trainee assessment. The provision of legal advice in cases of appeal and professional support by the resource and support office is emphasised.

Are Norwegian mentors failing to fail nursing students?

It is highly important that nursing students are well educated and become safe and competent practitioners. This article presents the findings from a quantitative study investigating if Norwegian mentors fail to fail nursing students not achieving the learning outcomes in clinical studies in the bachelor's programme in nursing, in addition to the factors influencing their decisions. A survey was conducted among nurse mentors in hospital- and home-based nursing care in two Norwegian municipalities, and 561 nurses answered the questionnaire. The findings indicate that mentors sometimes fail to fail nursing students in clinical studies. Important factors influencing this decision were that the students did not put the patient's life at risk and that the mentors gave the student the benefit of the doubt. The mentors in our study thought that failing to fail students was not related to personal challenges and burdens. They also felt a lack of support from the educational institution. The findings suggest some future directions for nurse education programmes. The training of mentors, especially in managing failing students, and better support from liaison lecturers from the educational institution are important. It is also suggested that nurse education programmes together with mentors discuss the distinction between unsafe practices and learning outcomes for clinical studies.

Keeping mum in clinical supervision: private thoughts and public judgements

CONTEXT

The seemingly obvious claim that people prefer to keep mum about undesirable messages - termed 'the MUM effect' - was initially reported in the psychology literature in the 1970s. More recently, it has been discussed in contexts including performance appraisals and the reporting of unsuccessful projects in workplace settings, but only sparsely in educational ones. We wished to review the published literature on the MUM effect in order to understand the implications for clinical assessment.

METHODS

We performed a narrative literature review on the MUM effect and clustered findings together into three themes: those that describe what MUM behaviours look like, those that explore potential reasons for the MUM effect and those that consider factors that can influence MUM behaviours.

RESULTS

This paper summarises the extensive literature on the MUM effect, including its manifestations and modifiers and discusses how the effect may be used to consider issues faced by many clinical supervisors faced with delivering 'negative' assessment messages to trainees.

DISCUSSION

We suggest, that as a pervasive phenomenon, the MUM effect can both help to explain the difficulties that some assessors face when delivering undesirable messages (including feedback or ratings) and offer new insights in how to deal with such issues.

Clinical Instructors' Experiences Working with and Assessing Students Who Perform below Expectations in Physical Therapy Clinical Internships

Purpose: Clinical education is an integral component of the curriculum of all physical therapy (PT) entry-to-practice programmes in Canada. The literature indicates that working with and assessing students performing below expectations (SPBE) can be procedurally and emotionally difficult. Our study aimed to explore the experiences of clinical instructors (CIs) and the decision-making process involved when supervising SPBE in PT. Method: A total of 19 in-depth, semi-structured interviews were conducted with CIs, transcribed, and coded using qualitative thematic analysis. Results: Four factors appeared to be important for CIs when they were deciding how to assess SPBE: (1) features of student performance, (2) factors related to the CIs, (3) academic and clinical facility influencers, and (4) strategies and available resources. Concerns about safety and professional behaviour, a student's clinical reasoning skills, and a lack of progression were key factors that CIs considered in recommending a final grade. CIs were more likely to recommend a failing grade if there was a series of repeated incidents rather than an isolated incident. Conclusions: We make several recommendations for the student, CI, and facilities to consider to better support and facilitate the process of working with SPBE in PT clinical education.

The challenges of detecting progress in generic competencies in the clinical setting

CONTEXT

Competency-based medical education has spurred the implementation of longitudinal workplace-based assessment (WBA) programmes to track learners' development of competencies. These hinge on the appropriate use of assessment instruments by assessors. This study aimed to validate our assessment programme and specifically to explore whether assessors' beliefs and behaviours rendered the detection of progress possible.

METHODS

We implemented a longitudinal WBA programme in the third year of a primarily rotation-based clerkship. The programme used the professionalism mini-evaluation exercise (P-MEX) to detect progress in generic competencies. We used mixed methods: a retrospective psychometric examination of student assessment data in one academic year, and a prospective focus group and interview study of assessors' beliefs and reported behaviours related to the assessment.

RESULTS

We analysed 1662 assessment forms for 186 students. We conducted interviews and focus groups with 21 assessors from different professions and disciplines. Scores were excellent from the outset (3.5-3.7/4), with no meaningful increase across blocks (average overall scores: 3.6 in block 1 versus 3.7 in blocks 2 and 3; F = 8.310, d.f. 2, p < 0.001). The main source of variance was the forms (47%) and only 1% of variance was attributable to students, which led to low generalisability across forms (Eρ²  = 0.18). Assessors reported using multiple observations to produce their assessments and were reluctant to harm students by consigning anything negative to writing. They justified the use of a consistent benchmark across time by citing the basic nature of the form or a belief that the 'competencies' assessed were in fact fixed attributes that were unlikely to change.

CONCLUSIONS

Assessors may purposefully deviate from instructions in order to meet their ethical standards of good assessment. Furthermore, generic competencies may be viewed as intrinsic and fixed rather than as learnable. Implementing a longitudinal WBA programme is complex and requires careful consideration of assessors' beliefs and values.

A dietetic clinical educator enhances the experience and assessment of clinical placement

AIM

The aim of this study was to evaluate the impact of a Clinical Educator model on the learning experience and environment for students, preceptors and managers.

METHODS

A Clinical Educator position was established for the 10-week dietetic clinical placement at Edith Cowan University. The Clinical Educator was responsible for overseeing the placement and assisting in the supervision of students. A qualitative descriptive approach using focus groups with purposive sampling explored the research question. Students (n = 10), preceptors (n = 21) and managers (n = 3) participated in separate focus groups. Data were thematically analysed with consideration given to participant and focus group commonalities and differences.

RESULTS

The findings revealed that the Clinical Educator (i) reduced the logistical burden of student placements and improved time efficiency; (ii) facilitated student assessment within a programme of assessment; (iii) was uniquely positioned to provide support and enhance student confidence; and (iv) enhanced capacity to manage underperforming and challenging students.

CONCLUSIONS

The Clinical Educator model increased student confidence, facilitated quality assessment and supported the management of underperforming students. This was achieved by reducing the burden of clinical placements, facilitating effective and timely communication between stakeholders and supporting the establishment of meaningful relationships which enriched learning. The results highlight the importance of the people involved in placement to facilitate a positive student learning environment and high quality assessment.

Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth From Ten Pediatrics Residency Programs

PURPOSE

This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workload-unadjusted versus workload-adjusted scores.

METHOD

Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workload-adjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time.

RESULTS

On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workload-unadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time.

CONCLUSIONS

Incorporating a measure of observer-reported workload could improve the measurement properties and the ability to interpret WBA scores.

Silent Witnesses: Faculty Reluctance to Report Medical Students' Professionalism Lapses

PURPOSE

Assessing students' professionalism is a critical component of medical education. Nonetheless, faculty reluctance to report professionalism lapses remains a significant barrier to the effective identification, management, and remediation of such lapses. The authors gathered information from faculty who supervise medical students to better understand their perceived barriers to reporting.

METHOD

In 2015-2016, data were collected using a group concept mapping methodology, which is an innovative, asynchronous, structured mixed-methods approach using qualitative and quantitative measures to identify themes characterizing faculty reluctance to report professionalism lapses. Participants from four U.S. and Canadian medical schools brainstormed, sorted, and rated statements about perceived barriers to reporting. Multidimensional scaling and hierarchical cluster analyses were used to analyze these data.

RESULTS

Of 431 physicians invited, 184 con-tributed to the brainstorming task (42.7%), 48 completed the sorting task (11.1%), and 83 completed the rating task (19.3%). Participants identified six barriers or themes to reporting lapses. The themes "uncertainty about the process," "ambiguity about the 'facts,'" "effects on the learner," and "time constraints" were rated highest as perceived barriers. Demographic subgroup analysis by gender, years of experience supervising medical students, years since graduation, and practice discipline revealed no significant differences (P > .05).

CONCLUSIONS

The decision to report medical students' professionalism lapses is more complex and nuanced than a binary choice to report or not. Faculty face challenges at the systems level and individual level. The themes identified in this study can be used for faculty development and to improve processes for reporting students' professionalism lapses.

Qualitative Pilot Study of Veterinarians' Perceptions of and Experiences with Counseling about Dog Walking in Companion-Animal Practice in Southern Ontario

Dog walking is beneficial to dogs and their owners, and it supports One Health, an initiative devoted to improving the health of animals and humans. Despite the benefits of dog walking, many dogs and their owners are not engaging in adequate dog-walking exercise. One way to encourage dog walking is for veterinarians to counsel owners on dog walking during veterinary appointments. This pilot study used individual in-person or telephone interviews to understand veterinarians' perceptions of and experiences with counseling about dog walking in companion animal practice. Seventeen veterinarians licensed to provide canine care in Ontario participated in an interview in 2015. Thematic analysis was used to analyze the interview transcripts. Participants thought dog walking was important for both dogs and owners and discussed physical, social, and mental benefits of dog walking. Participants described barriers to dog-walking counseling including dog owners' lack of compliance or physical condition, dogs' behavior, and lack of time or low priority during appointments. Also, participants discussed how dog walking was not addressed in their professional veterinary education. These findings can be used to educate veterinary students and practicing veterinarians on the value of dog walking for people and dogs and the barriers they may face when counseling clients about dog walking.

Exploring mentors' interpretation of terminology and levels of competence when assessing nursing students: An integrative review

OBJECTIVES

The purpose of this integrative review is to evaluate the empirical and theoretical literature on the challenges mentors face in interpreting and assessing levels of competence of student nurses in clinical practice.

DESIGN

An integrative review of the literature.

DATA SOURCES

An extensive and systematic literature search was conducted covering the period 1986-September 2016 across twelve databases covering health and education related publications. Grey literature was searched from wide relevant sources.

REVIEW METHODS

Sources were eligible for review when they referred to mentor's interpretation or assessment of student nurses' level of competence in practice settings. Methodological rigor of the included studies was evaluated with the Mixed Methods Appraisal Tool.

RESULTS

After screening 1951 records by titles, abstracts and full text, 8 were selected for review. The methodological quality of the studies was moderate. The studies reported: Accurate and fair assessment of students is impeded by a lack of transparent and explicit criteria.

CONCLUSIONS

There is a need to establish a transparent and common language to distinguish between and facilitate interpretation of different levels of competence. Well-designed rubrics might offer the solution to the challenges faced in practice-based assessment and necessitates further research into their use.

What do quantitative ratings and qualitative comments tell us about general surgery residents' progress toward independent practice? Evidence from a 5-year longitudinal cohort

BACKGROUND

This study examines the alignment of quantitative and qualitative assessment data in end-of-rotation evaluations using longitudinal cohorts of residents progressing throughout the five-year general surgery residency.

METHODS

Rotation evaluation data were extracted for 171 residents who trained between July 2011 and July 2016. Data included 6069 rotation evaluations forms completed by 38 faculty members and 164 peer-residents. Qualitative comments mapped to general surgery milestones were coded for positive/negative feedback and relevance.

RESULTS

Quantitative evaluation scores were significantly correlated with positive/negative feedback, r = 0.52 and relevance, r = -0.20, p < .001. Themes included feedback on leadership, teaching contribution, medical knowledge, work ethic, patient-care, and ability to work in a team-based setting. Faculty comments focused on technical and clinical abilities; comments from peers focused on professionalism and interpersonal relationships.

CONCLUSIONS

We found differences in themes emphasized as residents progressed. These findings underscore improving our understanding of how faculty synthesize assessment data.

Struggle and failure on clinical placement: a critical narrative review

BACKGROUND

Clinical placements are crucial to the development of skills and competencies in speech-language pathology (SLP) education and, more generally, a requirement of all health professional training programmes. Literature from medical education provides a context for understanding how the environment can be vital to all students' learning. Given the increasing costs of education and demands on health services, students who struggle or fail on clinical placement place an additional burden on educators. Therefore, if more is known or understood about these students and their experience in relation to the clinical learning environment, appropriate strategies and support can be provided to reduce the burden. However, this literature does not specifically explore marginal or failing students and their experience.

AIMS

To review existing research that has explored failing and struggling health professional students undertaking clinical placements and, in particular, SLP students.

METHODS & PROCEDURES

A critical narrative review was undertaken. Three electronic databases, ProQuest, CINAHL and OVID (Medline 1948-), were searched for papers exploring marginal and failing students in clinical placement contexts across all health professions, published between 1988 and 2017. Data were extracted and examined to determine the breadth of the existing research, and publications were critically appraised and major research themes identified.

MAIN CONTRIBUTION

Sixty-nine papers were included in the review. The majority came from medicine and nursing in the United States and United Kingdom, with other allied health disciplines less well represented. The review identified key themes with the majority of papers focused on identification of at risk students and support and remediation. The review also highlighted the absence of literature relating to the student voice and in the allied health professions.

CONCLUSIONS & IMPLICATIONS

This review highlighted the limited research related to failing/struggling student learning in clinical contexts, and only a handful of papers have specifically addressed marginal or failing students in allied health professions. The complexity of interrelated factors in this field has been highlighted in this review. Further research needs to include the student's voice to develop greater understanding and insights of struggle and failure in clinical contexts.

Exploring the influence of context on feedback at medical school: a video-ethnography study

Feedback in medical education is complicated by the multiple contexts within which learning occurs. However, feedback research in medical education has typically focused on information provided by tutors to students with limited exploration of the influence of context. This research seeks to address this gap by exploring the influence of multiple contexts upon feedback processes. Employing video-ethnography methodology we explored feedback in two common contexts for medical student learning: the simulated clinical environment and the medical workplace. Learning and teaching sessions were filmed in each of these contexts, capturing diverse feedback processes. Data were analysed for key themes using a Framework Analysis approach and similarities and differences between the two contexts identified. In total 239 distinct feedback episodes across 28 different teaching and learning sessions were captured, with feedback processes relating to the patient, practice, educational and institutional contexts observed. In this paper, we concentrate on key similarities and differences in feedback processes between the two contexts with respect to six themes: feedback interlocutors, interlocutor positioning, feedback types, feedback foci, feedback styles and feedback milieu. We argue that feedback is inextricably linked to the multiple contexts in which feedback is enacted. It is only by exploring these contextual influences that feedback can be understood more fully. With such understanding we should be better placed to develop interventions capable of improving the long elusive experience of successful feedback.

Exploring trainer and trainee emotional talk in narratives about workplace-based feedback processes

Emotion characterises learners' feedback experiences. While the failure-to-fail literature suggests that emotion may be important, little is known about the role of emotion for educators. Secondary analyses were therefore conducted on data exploring 110 trainers' and trainees' feedback experiences. Group and individual narrative interviews were conducted across three UK sites. We analysed 333 narratives for emotional talk using textual analysis: Linguistic Inquiry and Word Count. Furthermore, thematic framework analysis was conducted on the trainer narratives to explore aspects of feedback processes that are emotional. An additional in-depth little 'd' discourse analysis was conducted on selected trainer narratives to enable us to explore the complex relationship between the whats (reported events) and the hows (emotional talk). Trainer narratives did not differ significantly in positive or negative emotional talk from trainee narratives. By exploring the interplay of the whats and the hows, several aspects of feedback processes were identified as potentially emotional for trainers including trainers being concerned about upsetting learners and worried about patient safety. This was illustrated through numerous linguistic devices to establish emotional tone such as metaphoric talk and laughter. These findings suggest that feedback processes can be emotional for trainers. It highlights the need to better understand the 'filter' of emotion for trainers but also to better understand how emotion plays a role in feedback as a complex social process.

Incorporating patient partner scores into high stakes assessment: an observational study into opinions and attitudes

BACKGROUND

Volunteer patients (also known as patient partners (PPs)) play a vital role in undergraduate healthcare curricula. They frequently take part in objective structured clinical examinations (OSCE) and rate aspects of students' performance. However, the inclusion and weighting of PP marks varies, while attitudes and opinions regarding how (and if) they should contribute towards the pass/fail outcome are uncertain.

METHODS

A prospective observational study was conducted to explore beliefs of PPs regarding inclusion of their scores in a high stakes undergraduate OSCE in a single UK medical school. All PPs delivering components of the local MBChB curriculum were asked to participate in the questionnaire study. Quantitative and qualitative data were analysed using descriptive statistics and framework analysis respectively.

RESULTS

Fifty out of 160 (31% response rate) PPs completed the questionnaire; 70% had participated in a final year OSCE. Thirty (60%) felt their marks should be incorporated into a student's overall score, while 28% were uncertain. The main reasons for inclusion were recognition of the patient perspective (31%) and their ability to assess attitudes and professionalism (27%), while reasons against inclusion included lack of PP qualification/training (18%) and concerns relating to consistency (14%). The majority of PPs were uncertain what proportion of the total mark they should contribute, although many felt that 5-10% of the total score was reasonable. Most respondents (70%) felt that globally low PP scores should not result in an automatic fail and many (62%) acknowledged that prior to mark inclusion, further training was required.

CONCLUSION

These data show that most respondents considered it reasonable to "formalise their expertise" by contributing marks in the overall assessment of students in a high stakes OSCE, although what proportion they believe this should represent was variable. Some expressed concerns that using marks towards progress decisions may alter PP response patterns. It would therefore seem reasonable to compare outcomes (i.e. pass/fail status) using historical data both incorporating and not incorporating PP marks to evaluate the effects of doing so. Further attention to existing PP training programmes is also required in order to provide clear instruction on how to globally rate students to ensure validity and consistency.

The Hidden Value of Narrative Comments for Assessment: A Quantitative Reliability Analysis of Qualitative Data

PURPOSE

In-training evaluation reports (ITERs) are ubiquitous in internal medicine (IM) residency. Written comments can provide a rich data source, yet are often overlooked. This study determined the reliability of using variable amounts of commentary to discriminate between residents.

METHOD

ITER comments from two cohorts of PGY-1s in IM at the University of Toronto (graduating 2010 and 2011; n = 46-48) were put into sets containing 15 to 16 residents. Parallel sets were created: one with comments from the full year and one with comments from only the first three assessments. Each set was rank-ordered by four internists external to the program between April 2014 and May 2015 (n = 24). Generalizability analyses and a decision study were performed.

RESULTS

For the full year of comments, reliability coefficients averaged across four rankers were G = 0.85 and G = 0.91 for the two cohorts. For a single ranker, G = 0.60 and G = 0.73. Using only the first three assessments, reliabilities remained high at G = 0.66 and G = 0.60 for a single ranker. In a decision study, if two internists ranked the first three assessments, reliability would be G = 0.80 and G = 0.75 for the two cohorts.

CONCLUSIONS

Using written comments to discriminate between residents can be extremely reliable even after only several reports are collected. This suggests a way to identify residents early on who may require attention. These findings contribute evidence to support the validity argument for using qualitative data for assessment.

Descriptors for unprofessional behaviours of medical students: a systematic review and categorisation

BACKGROUND

Developing professionalism is a core task in medical education. Unfortunately, it has remained difficult for educators to identify medical students' unprofessionalism, because, among other reasons, there are no commonly adopted descriptors that can be used to document students' unprofessional behaviour. This study aimed to generate an overview of descriptors for unprofessional behaviour based on research evidence of real-life unprofessional behaviours of medical students.

METHODS

A systematic review was conducted searching PubMed, Ebsco/ERIC, Ebsco/PsycINFO and Embase.com from inception to 2016. Articles were reviewed for admitted or witnessed unprofessional behaviours of undergraduate medical students.

RESULTS

The search yielded 11,963 different studies, 46 met all inclusion criteria. We found 205 different descriptions of unprofessional behaviours, which were coded into 30 different descriptors, and subsequently classified in four behavioural themes: failure to engage, dishonest behaviour, disrespectful behaviour, and poor self-awareness.

CONCLUSIONS

This overview provides a common language to describe medical students' unprofessional behaviour. The framework of descriptors is proposed as a tool for educators to denominate students' unprofessional behaviours. The found behaviours can have various causes, which should be explored in a discussion with the student about personal, interpersonal and/or institutional circumstances in which the behaviour occurred. Explicitly denominating unprofessional behaviour serves two goals: [i] creating a culture in which unprofessional behaviour is acknowledged, [ii] targeting students who need extra guidance. Both are important to avoid unprofessional behaviour among future doctors.

Justice and care: decision making by medical school student promotions committees

CONTEXT

The function of medical school entities that determine student advancement or dismissal has gone largely unexplored. The decision making of 'academic progress' or student promotions committees is examined using a theoretical framework contrasting ethics of justice and care, with roots in the moral development work of theorists Kohlberg and Gilligan.

OBJECTIVES

To ascertain promotions committee members' conceptualisation of the role of their committee, ethical orientations used in member decision making, and student characteristics most influential in that decision making.

METHODS

An electronic survey was distributed to voting members of promotions committees at 143 accredited allopathic medical schools in the USA. Descriptive statistics were calculated and data were analysed by gender, role, institution type and class size.

RESULTS

Respondents included 241 voting members of promotions committees at 55 medical schools. Respondents endorsed various promotions committee roles, including acting in the best interest of learners' future patients and graduating highly qualified learners. Implementing policy was assigned lower importance. The overall pattern of responses did not indicate a predominant orientation toward an ethic of justice or care. Respondents indicated that committees have discretion to take individual student characteristics into consideration during deliberations, and that they do so in practice. Among the student characteristics with the greatest influence on decision making, professionalism and academic performance were paramount. Eighty-five per cent of participants indicated that they received no training.

CONCLUSIONS

Promotions committee members do not regard orientations of justice and care as being mutually exclusive and endorse an array of statements regarding the committee's purpose that may conflict with one another. The considerable variance in the influence of student characteristics and the general absence of committee member training indicate a need for clear delineation of the medical profession's priorities in terms of justice and care, and of the specific student characteristics that should factor into deliberations.

The qualitative orientation in medical education research

Qualitative research is very important in educational research as it addresses the "how" and "why" research questions and enables deeper understanding of experiences, phenomena and context. Qualitative research allows you to ask questions that cannot be easily put into numbers to understand human experience. Getting at the everyday realities of some social phenomenon and studying important questions as they are really practiced helps extend knowledge and understanding. To do so, you need to understand the philosophical stance of qualitative research and work from this to develop the research question, study design, data collection methods and data analysis. In this article, I provide an overview of the assumptions underlying qualitative research and the role of the researcher in the qualitative process. I then go on to discuss the type of research objectives which are common in qualitative research, then introduce the main qualitative designs, data collection tools, and finally the basics of qualitative analysis. I introduce the criteria by which you can judge the quality of qualitative research. Many classic references are cited in this article, and I urge you to seek out some of these further reading to inform your qualitative research program.

'Seeking authorization': a grounded theory exploration of mentors' experiences of assessing nursing students on the borderline of achievement of competence in clinical practice

AIM

To develop a substantive theoretical explanation of how mentors make sense of their experiences where nursing students are on the borderline of achievement of competence in clinical practice.

BACKGROUND

The reluctance of Registered Nurse mentors to fail nursing students in clinical practice despite concerns about competence remains a contemporary issue in international healthcare education. Mentors' assessment decisions have considerable impact for a variety of key stakeholders, not least for students in these circumstances.

DESIGN

Grounded theory qualitative study.

METHODS

Phase one involved 20 individual semi-structured interviews with nurse mentors in one United Kingdom National Health Service Health Board (July-October 2009). Phase two included eight individual semi-structured interviews and seven focus groups with mentors and practice educators (n = 38) in four further Health Boards (June 2011-February 2012). Data were analysed using open, axial and selective coding consistent with grounded theory method.

FINDINGS

Three categories 'the conundrum of practice competence,' 'the intensity of nurturing hopefulness,' and 'managing assessment impasse,' led to the study's substantive theoretical explanation - 'Seeking authorization: Establishing collective accountability for mentorship.' This demonstrates how mentors are dependent on key sources of support and feedback to validate their assessment decision-making, notwithstanding substantial personal, professional and organizational pressures.

CONCLUSION

We conclude that management of borderline assessment situations is considerably developed by recognition of the authorizing effects of a wider community of assessors. Consequently, we identify the personal, professional and organizational implications involved in the preparation, support and regulation of mentors specifically during borderline assessment circumstances.

Relating professionalism and conscientiousness to develop an objective, scalar, proxy measure of professionalism in anaesthetic trainees

BACKGROUND

The concept of professionalism is complex and subjective and relies on expert judgements. Currently, there are no existing objective measures of professionalism in anaesthesia. However, it is possible that at least some elements of professionalism may be indicated by objective measures. A number of studies have suggested that conscientiousness as a trait is a significant contributor to professionalism.

METHODS

A 'Conscientiousness Index' (CI) was developed by collation of routinely collected data from tasks expected to be carried out by anaesthetic trainees such as punctual submission of holiday and 'not-on-call' requests, attendance at audit meetings, timely submission of completed appraisal documentation and sickness/absence notifications. The CI consists of a sum of points deducted from a baseline of 50 for non-completion of these objective and measurable behaviours related to conscientiousness. This was correlated with consultants' formal and informal subjective measures of professionalism in those trainees. Informal, subjective measures of professionalism consisted of a 'Professionalism Index' (PI). The PI consisted of a score developed from consultants' expert, subjective views of professionalism for those trainees. Formal, subjective measures of professionalism consisted of a score derived from comments made by consultants in College Tutor feedback forms on their views on the professionalism of those trainees (College Tutor feedback; CT). The PI and CT scores were correlated against the CI using a Pearson or Spearman correlation coefficient.

RESULTS

There was a negative, but not statistically significant, relationship between the CI and formal, subjective measures of professionalism; CT scores (r = -0.341, p = 0.06), but no correlation between CI and consultants informal views of trainees' professionalism; the PI scores (r _s  = -0.059, p = 0.759).

CONCLUSIONS

This may be due the 'failure to fail' phenomenon due to the high stakes nature of raising concerns of professionalism in postgraduate healthcare professionals or may be that the precision of the tool may be insufficient to distinguish between trainees who generally show highly professional behaviour. Future development of the tool may need to include more of the sub-facets of conscientiousness. Independently of a relationship with the construct of professionalism, a measure of conscientiousness might be of interest to future employers.

In-training assessments: 'The difficulty is trying to balance reality and really tell the truth'

BACKGROUND

In-training assessments (ITA) aim to evaluate trainees' progress and give valuable feedback on their performance. Many factors can affect supervisors during their completion of assessments and these can influence the final results recorded.

METHODS

This is the second part of a study of supervisors of the Australasian College of Dermatologists (ACD) and presents the qualitative data on their opinions of the ACD ITA process and the influences on their ITA ratings.

RESULTS

Supervisors noted the benefits of this assessment tool, together with many limitations. Potential influences upon supervisor ratings included the relationship between the supervisor and trainee and the level of honesty in completing and delivery of the assessment.

CONCLUSIONS

Many factors influence supervisors in the completion of the ITA. These include the impact of interpersonal relationships and concerns about the consequences of delivering a negative assessment, which sometimes lead supervisors to modify the assessment they deliver to the trainee. Further research is needed into honesty in assessment judgements.