Do trainees value feedback in case-based discussion assessments?

Factors that may improve paediatric workplace-based assessments: an exploratory study

OBJECTIVES

To establish if paediatric trainees are satisfied with the current workplace-based assessment (WBA) process. To identify factors that contribute both positively and negatively to the educational experience during WBAs. To find out if trainees and their supervisors experience any challenges conducting WBAs. To establish potential ways to improve future assessments.

DESIGN

Qualitative semistructured interviews.

SETTING

Participants included fifteen trainees (ST1-8) in general paediatric and subspecialty posts and four consultants or equivalent across five hospital sites in the Thames Valley Deanery. All participants had regular exposure to WBAs.

INTERVENTIONS

Interviews were undertaken between June 2020 and January 2021 via video link. Data collection and analysis were conducted iteratively using constant comparison until theoretical sufficiency was achieved.

MAIN OUTCOME MEASURE

Using Constructivist Grounded Theory, a theoretical framework, grounded in the data, was developed that depicted the core elements that should be present to optimise WBAs.

RESULTS

A number of key components were reported to affect the educational value of WBAs. A positive departmental culture towards education and training is essential. Chosen cases should be challenging, and direct observation or in-depth discussion, depending on the assessment type, is fundamental. Timely constructive feedback and immediate completion of the assessment form are also imperative.

CONCLUSION

Some trainees experienced WBAs where these key components aligned, but many did not, and this negatively affected their learning. Three main challenges or future targets for further improvements include increasing time, improving training and optimising technology.

The usefulness of work-based assessments in higher surgical training: A systematic review

BACKGROUND

Work-based assessments including procedure based assessments, case based discussions, clinical evaluation exercises and direct observation of procedural skills are used in Higher Surgical Training Program. This systematic review aims to investigate the trainer and trainee's perception of the usefulness of workplace based assessments in Higher Surgical Training Programme.

MATERIALS AND METHODS

Embase, MEDLINE and PubMed databases were searched for relevant studies published up to Jan 15, 2021. The following search terms were used: procedure based assessments, case based discussions, clinical evaluation exercises, direct observation of procedural skills, Higher Surgical Training Program (with and without their abbreviations), surgical training, formative assessment. Usefulness was analysed according to van der Vleuten's utility formula, which is the product of educational impact, validity, reliability, acceptability, cost-effectiveness and feasibility.

RESULTS

23 studies were included; Six on procedure based assessments, two on case based discussions, one on mini clinical evaluation exercises, three on direct observation of procedural skills and eleven on multiple methods. Overall, procedure based assessments had positive Kirkpatrick level 1 or 2 impact were valid, reliable and acceptable. Mini clinical evaluation exercises showed positive Kirkpatrick level 1 satisfaction in trainees and trainers and were feasible. Case based discussions were shown to have both positive and negative Kirkpatrick levels 1 and 2 impact and were rated valid and reliable. Direct observation of procedural skills were valid with favourable Kirkpatrick level 1 impact with more usage and modification of the forms. Multiple methods as used in the intercollegiate surgical curriculum programme portfolio was found to be less encouraging in achieving positive higher level of educational impact. The barriers included lack of time, lack of faculty development and lack of engagement.

CONCLUSION

It is important that work-based assessments are used properly, so that trainees get the maximum benefit from them. None of the studies included in our systematic review demonstrated an impact on behaviour or society (Kirkpatrick level 3 and 4), towards which future research should be directed to.

Can case-based discussions in a group setting be used to assess residents' clinical skills?

OBJECTIVES

The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept.

METHODS

This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach.

RESULTS

Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge.

CONCLUSIONS

The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.

Case-Based Discussion in United Kingdom General Practice Training: A Critical Analysis

Case-based discussion (CbD) is a form of workplace-based assessment to assess the progress of learning in general practice trainees in the United Kingdom. We aim to identify the need and rationale behind CbD. The usefulness of CbD in the trainee’s learning will be analyzed with the help of well-recognized parameter such as utility equation. It will also be considered whether the assessment delivers what it is supposed to in actual practice. The current pitfalls will be identified with suggestions for potential improvements.

Workplace-based assessments in postgraduate medical education: A hermeneutic review

OBJECTIVES

Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education.

METHOD

This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached.

RESULTS

Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline.

CONCLUSIONS

The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

Assessment methods in respiratory medicine training in Europe: current status and future needs

There is a perceived need for harmonisation of training standards in medicine across Europe. Assessment methodology is a fundamental part of this harmonisation. Assessment may be: summative, measuring competency at the end of training, usually in a pass/fail mode; or formative, assessing strengths and weaknesses during training with a view to optimising performance. A survey of experienced respiratory medicine trainers in Europe was undertaken to discover the formative and summative assessment methods being used, and to explore the use of workplace-based assessments (WBAs). Structured interviews were sought with experienced trainers in adult and paediatric respiratory medicine. 35 trainers from 22 (71%) out of 31 countries were interviewed. The number of types of summative assessments required at the end of training varied from 0 to 4 including clinical (25%), written (44%) and oral (47%) examinations. Four respondents required a research thesis as a proof of clinical competence. WBA was not commonly used. Only 14% of respondents reported using a formal case-based discussion, 20% used a form of multisource feedback and 25% described some form of formal assessment of procedural skills. However, 77% of all respondents expressed a wish to have access to case-based discussion and 72% wanted to have a facility for multisource feedback. The majority also wanted training in the use of these tools. Almost half of the respondents had received no formal training in educational supervision and 80% of all respondents expressed a wish to receive such training. The findings suggest that there is no adequate process of ensuring uniform standards for specialist accreditation in Europe, and demonstrate a need and desire among trainers for more WBA tools and training in their use to be made available.

Workplace-based assessment: how to use case-based discussion as a formative assessment

Workplace-based assessments are increasingly used as a way of gaining insight into clinician performance in real-life situations. Although some can be used to inform a summative (pass/fail) assessment, many have a much greater role in the formative assessment of trainees, and can be used as tools for teaching and training and in identifying the development needs of trainees. There is considerable variation between different European countries in the use of formative, workplace-based assessment, such as a structured case-based discussion (CbD), during training. This article gives an overview of how to use CbD as a formative assessment for higher specialist trainees, and gives access to a downloadable record form which can be used by trainers.

Case-based discussion is a structured method of formative assessment which has been found to be valuable by both trainees and supervisors. This article describes the process, and offers a proforma for supervisors who have no access to this facility.http://bit.ly/2HYkOVJ

Use of work-based assessments in postgraduate medical education

Work-based assessments are ubiquitous in postgraduate medical training in the UK. This article discusses the variety of these assessments and explores barriers to their use and solutions for improving the educational value of these tools for adult learners. The focus should be on feedback and learning rather than assessment, and this may promote discussion of more challenging scenarios where the opportunity for learning is greater. Mobile devices may help reduce the administrative, geographical and time constraints of completing work-based assessments.

Trainees' perspectives of assessment messages: a narrative systematic review

OBJECTIVES

This study was designed as a narrative systematic literature review of medical specialist trainees' perspectives of the assessment messages they receive in the context of clinical performance assessments. The aim of the study was to determine if trainees value the information they receive through the formats designed to promote their development and, if not, the reasons for this.

METHODS

The authors searched the ERIC, EMBASE, Ovid MEDLINE and PsycINFO databases for articles published up to 16 June 2018 that present original data on trainees' perspectives of the assessment messages they receive in the context of work-based assessments (WBAs) and in-training assessments (ITAs) used within their training programmes. All authors screened 938 abstracts and 139 full-text articles were assessed after this. Descriptions of quantitative data and thematic analysis of qualitative data were used to present the opinions of trainees.

RESULTS

Thirty-three articles met the inclusion criteria. Twenty-six articles (79%) described trainees' perspectives in the context of WBA and the remaining articles referred to ITA formats. Wide-ranging opinions were reported. The analysis categorised these into three themes: trainees value developmental assessment messages; trainees become disengaged when assessment messages are not developmental, and trainees' views depend on the environment, the assessor and themselves. Some trainees reported that the assessment messages were valuable and provided input on their performance to guide their development, but many disagreed. In particular, the trainee's own level of engagement with the assessments influenced his or her perspectives on the messages received.

CONCLUSIONS

Trainees do not universally perceive that clinical performance assessments provide them with the valuable developmental input on their performance they were designed to do. Factors related to the environment, the assessor and themselves influence their perspectives.

Twelve tips on writing a discussion case that facilitates teaching and engages learners

The authors share twelve practical tips on writing a case that engages learners in active learning and discussion. They first advise that, during the initial preparation of the case, authors should (1) identify the case goals and objectives, and (2) identify the level of the learners. When writing the case, authors should (3) use active and colorful language; (4) use patients' own descriptions rather than medical language; (5) allow the learners to interpret data themselves; (6) allow for natural discovery rather than presenting information chronologically; and (7) be realistic about interruptions in patient care. In addition, case authors should pay attention to methods that enhance discussion by (8) creating barriers to diagnostic or treatment options; (9) promoting questions and discussion over answers; (10) using cues to assure discussion flow and knowledge exploration; and (11) omitting details or inserting informational distractors. Finally, well-crafted questions are essential during the case presentation to engage learners in higher-order thinking; and to (12) stimulate curiosity and reflection.

Impact of structured verbal feedback module in medical education: A questionnaire- and test score-based analysis

Introduction:

Feedback is a divalent bond between the supplier (teacher) and the recipient (student). The strength of the bond depends on the instructional design of the feedback. Feedback is central to medical education in promoting self-directed learning in students. In the present study, a structured verbal feedback module was prepared, implemented, and evaluated.

Methods:

The study was done on 280 students from four consecutive batches (2011 to 2014) of the 1^st year MBBS students exposed to different types and modes of feedback. Analysis was done using student feedback questionnaire for the perception of students to verbal feedback. Quantitative analysis using post hoc test and ANOVA for the impact of type of feedback (verbal or written) and effect of modes (individual or group) of verbal feedback on test score performance were done.

Result:

In this study, ≥95% of the students preferred verbal feedback of both positive and negative attributes in student questionnaires. It was observed that verbal feedback sessions made a difference of up to 2–2.4 grade points in the mean score of batch when compared to the written feedback. The initial mean test score (T1) of 2011 + 2012 and 2013 + 2014 was not statistically significant (P = 0.113). But, in all subsequent tests (T2, T3, and T4), there was a statistically significant difference in the mean test scores (P = 0.000).

Conclusion:

(1) Students prefer verbal one-to-one feedback over written feedback. (2) Verbal feedback changes learning process and causes sustained improvement in learning strategies.

Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings

Workplace based assessments (WBAs) are now commonplace in postgraduate medical training. User acceptability and engagement is essential to the success of any medical education innovation. To this end, possessing an insight into trainee and trainer perceptions towards WBAs will help identify the major problems, permitting strategies to be introduced to improve WBA implementation. A review of literature was performed to identify studies examining trainee and trainer perceptions towards WBAs. Studies were excluded if non-English or sampling a non-medical/dental population. The identified literature was synthesised for the purpose of this critical narrative review. It is clear that there is widespread negativity towards WBAs in the workplace. This has negatively impacted on the effectiveness of WBA tools as learning aids. This negativity exists in trainees but also to an extent in their trainers. Insight gained from the literature reveals three dominant problems with WBA implementation: poor understanding as to the purpose of WBAs; insufficient time available for undertaking these assessments; and inadequate training of trainers. Approaches to addressing these three problems with WBA implementation are discussed. It is likely that a variety of solutions will be required. The prevalence of negativity towards WBAs is substantial in both trainees and trainers, eroding the effectiveness of learning that is consequent upon them. The educational community must now listen to the concerns being raised by the users and consider the range of strategies being proposed to improve the experiences of trainees, and their trainers.

Using patients' charts to assess medical trainees in the workplace: a systematic review

OBJECTIVES

The objective of this review is to summarize and critically appraise existing evidence on the use of chart stimulated recall (CSR) and case-based discussion (CBD) as an assessment tool for medical trainees.

METHODS

Medline, Embase, CINAHL, PsycINFO, Educational Resources Information Centre (ERIC), Web of Science, and the Cochrane Central Register of Controlled Trials were searched for original articles on the use of CSR or CBD as an assessment method for trainees in all medical specialties.

RESULTS

Four qualitative and three observational non-comparative studies were eligible for this review. The number of patient-chart encounters needed to achieve sufficient reliability varied across studies. None of the included studies evaluated the content validity of the tool. Both trainees and assessors expressed high level of satisfaction with the tool; however, inadequate training, different interpretation of the scoring scales and skills needed to give feedback were addressed as limitations for conducting the assessment.

CONCLUSION

There is still no compelling evidence for the use of patient's chart to evaluate medical trainees in the workplace. A body of evidence that is valid, reliable, and documents the educational effect in support of the use of patients' charts to assess medical trainees is needed.

Lessons learnt from a pilot of assessment for learning

A 12-month pilot was carried out on assessments for learning and assessments of learning as part of workplace-based assessments in postgraduate medical education. This was carried out in three regions and core medical trainees and higher specialty medical trainees participated. Focus groups and questionnaires were utilised to investigate the trainees' and trainers' experiences and perceptions of assessments for learning. The study demonstrated that the trainees and trainers perceived the newly introduced assessments for learning--supervised learning events (SLEs)--as learning tools. However, SLEs were often undertaken with no previous organisation and with no direct observation, regardless of the underlying purposes and methods of the WPBAs. There was a lack of, or delayed or non-specific, feedback following SLEs, which would have impeded its educational value. Trainee and trainer disengagement was one of the contributing factors. These findings are valuable in informing and facilitating future successful implementation of assessments for learning.

Utilisation and outcomes of case-based discussion in otolaryngology training

OBJECTIVES

To assess the reliability, validity and outcomes of Case-based Discussion (CBD) in otolaryngology training.

DESIGN

Retrospective database analysis.

SETTING

National electronic database.

PARTICIPANTS

North London otolaryngology trainees.

MAIN OUTCOME MEASURES

We tested the tool's reliability along with its capacity to denote trainee progress. A score was calculated (cS) and compared across core (CT) and specialty trainees (ST) at all levels. The number of items rated as "development required" (D) was also examined.

RESULTS

One thousand four hundred and fifty-six CBDs were submitted by 46 trainees from 2007 to 2013, averaging 13.6 per trainee per year. Items relating to knowledge, management and judgement were more popular (98% usage), and better predictors of cS compared to other parameters (rs: +0.74, +0.70 and +0.72, respectively). CBD was found to be reliable (Cronbach's α = 0.848) and highly sensitive (99%), yet not specific. cS was significantly higher in ST than CT (95.3% ± 0.6 versus 88.7% ± 1.3). pS showed a similar pattern (3.15 ± 0.4 versus 2.0 ± 0.05) (P < 0.001). cS and pS increased from CT1 to ST8 (rs: +0.60 and +0.34, respectively). The number of D-rated items decreased with increasing year of training.

CONCLUSION

Case-based discussion is a reliable and valid tool in otolaryngology training. It is highly sensitive but not specific. Trainees should be encouraged to use it at all levels.

Otolaryngology WBAs in the Wales Deanery: the first six years

‘The provision of excellent care for the surgical patient, delivered safely, is at the heart of the curriculum.’ 1 Workplace-based assessments (WBAs) are an integral part of the assessment component of the UK’s intercollegiate surgical curriculum. The curriculum is web-based and accessed through www.iscp.ac.uk .

A critical review of the core medical training curriculum in the UK: A medical education perspective

This paper represents a systematic evaluation of the Core Medical Training Curriculum in the UK. The authors critically review the curriculum from a medical education perspective based mainly on the medical education literature as well as their personal experience of this curriculum. They conclude in practical recommendations and suggestions which, if adopted, could improve the design and implementation of this postgraduate curriculum. The systematic evaluation approach described in this paper is transferable to the evaluation of other undergraduate or postgraduate curricula, and could be a helpful guide for medical teachers involved in the delivery and evaluation of any medical curriculum