Development and validation of a computer-based situational judgement test to assess medical students' communication skills in the field of shared decision making

Integrating Shared Decision-Making into Undergraduate Oncology Education: A Pedagogical Framework

The integration of shared decision-making (SDM) into undergraduate oncology education represents a critical evolution in medical pedagogy, reflecting the growing complexity and patient-centric focus of contemporary healthcare. This paper introduces a comprehensive pedagogical framework designed to embed SDM within the undergraduate medical curriculum, particularly in oncology, where the multiplicity of treatment options and their profound impact on patient life underscore the necessity of this approach. Grounded in a systematic literature review and aligned with established educational theories, this framework proposes twelve strategic approaches to cultivate future physicians proficient in both clinical acumen and patient-collaborative decision-making. The framework emphasizes real-world clinical experience, role-playing, case studies, and decision aids to deepen students' understanding of SDM. It advocates for the development of communication skills, ethical deliberation, and cultural competence, recognizing the multifaceted nature of patient care. The inclusion of patient narratives and evidence-based decision-making further enriches the curriculum, offering a holistic view of patient care. Additionally, the integration of digital tools within the SDM process acknowledges the evolving technological landscape in healthcare. The paper also addresses challenges in implementing this framework, such as curricular constraints and the need for educator training. It underscores the importance of continual evaluation and adaptation of these strategies to the dynamic field of medical education and practice. Overall, this comprehensive approach aims not only to enhance the quality of oncological care but also to prepare medical students for the complexities of modern medicine, where patient involvement in decision-making is both a necessity and an expectation.

The educational value of situational judgement tests (SJTs) when used during undergraduate medical training: A systematic review and narrative synthesis

INTRODUCTION

Situational judgement tests (SJTs) are a recognised assessment method for admission into medical school, selection into postgraduate training programs, and postgraduate competency assessment. More recently, however, SJTs have been used during undergraduate medical training (UMT). This systematic review identifies, describes, and appraises the evidence for SJTs in UMT to determine educational associations and outcomes.

METHODS

MEDLINE, EMBASE, ERIC, PsycINFO, SCOPUS, Web of Science, and grey literature were searched for original research studies evaluating SJTs implemented within UMT to 1 November 2022. Studies reporting evaluation outcomes were included. Narrative data syntheses were undertaken. Risk of Bias was appraised using the Quality in Prognosis Studies tool.

RESULTS

24 studies were included. National database-derived SJTs (n = 14) assessed against professionalism, postgraduate attainment, construct of medical degree, medical school admissions scores, personality attributes, and declaration of disability. In-house derived SJTs (n = 10) assessed against professionalism, clinical skills, and personality attributes. Most evidence evaluated and reported inverse SJT associations with professionalism and were moderate risk of bias.

CONCLUSION

SJTs may have utility for developing professional behaviours in medical students. However, further research testing SJT robustness, standard setting methodologies, and prospectively evaluating SJTs against objective outcome measures within the context of UMT is warranted.

Does it make sense to use written instruments to assess communication skills? Systematic review on the concurrent and predictive value of written assessment for performance

OBJECTIVES

To evaluate possible associations between learners' results in written and performance-based assessments of communication skills (CS), either in concurrent or predictive study designs.

METHODS

Search included four databases for peer-reviewed studies containing both written and performance-based CS assessment. Eleven studies met the inclusion criteria.

RESULTS

Included studies predominantly assessed undergraduate medical students. Studies reported mainly low to medium correlations between written and performance-based assessment results (Objective Structured Clinical Examinations or encounters with simulated patients), and gave correlation coefficients ranging from 0.13 to 0.53 (p < 0.05). Higher correlations were reported when specific CS, like motivational interviewing were assessed. Only a few studies gave sufficient reliability indicators of both assessment formats.

CONCLUSIONS

Written assessment scores seem to predict performance-based assessments to a limited extent but cannot replace them entirely. Reporting of assessment instruments' psychometric properties is essential to improve the interpretation of future findings and could possibly affect their predictive validity for performance.

PRACTICE IMPLICATIONS

Within longitudinal CS assessment programs, triangulation of assessment including written assessment is recommended, taking into consideration possible limitations. Written assessments with feedback can help students and trainers to elaborate on procedural knowledge as a strong support for the acquisition and transfer of CS to different contexts.

Development and Implementation of Situational Judgment Tests as an Evaluation Method for Training Oncology Physicians: Application in the KOKON-KTO Training

Situational judgment tests (SJTs) are often used in aptitude testing and present practice-specific challenges. Their implementation into online training programs provides the opportunity to assess learning progress and improve training quality. In this study, text-based SJTs for oncology physicians were developed, validated, and implemented into the KOKON-KTO training which uses a blended learning training format to teach oncology physicians how to consult cancer patients on complementary and integrative medicine (CIM). The SJT was implemented to measure the e-learning results. In the development and validation phase, a total of 15 SJTs (each SJT including 1 best choice answer based on training content and 4 distractors; 9 SJTs for oncologists and 6 SJTs for oncology gynecologists only) were developed by an interprofessional team (n=5) using real-case vignettes and applying an in-depth review process. Best answers were validated by experts (oncologists and oncology gynecologists) with experience in advising cancer patients on CIM. In the implementation and evaluation phase, SJTs were answered by KOKON-KTO training participants (n=19) pre- and post e-learning. Results were analyzed using descriptive measurements, item difficulties, and Cohen's d for effect size pre- and post-training. The experts (n=12, 49.8% gynecologists) agreed with best choice answers (69.4% for oncology gynecology; 81.5% for oncology) in 12 out of 15 SJTs. Comparing pre- and post-training scores, KOKON-KTO training participants were able to improve knowledge substantially (effect sizes for oncologists d=1.7; oncology gynecologists d= .71). Future studies need to increase the number of experts and SJTs in order to apply further psychometric measurements. As part of the KOKON-KTO study, this project is registered as DRKS00012704 on the "German Clinical Trials Register" (Date of registration: 28.08.2017).

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

What do we know about written assessment of health professionals' communication skills? A scoping review

OBJECTIVE

The aim of this scoping review was to investigate the published literature on written assessment of communication skills in health professionals' education.

METHODS

Pubmed, Embase, Cinahl and Psychnfo were screened for the period 1/1995-7/2020. Selection was conducted by four pairs of reviewers. Four reviewers extracted and analyzed the data regarding study, instrument, item, and psychometric characteristics.

RESULTS

From 20,456 assessed abstracts, 74 articles were included which described 70 different instruments. Two thirds of the studies used written assessment to measure training effects, the others focused on the development/validation of the instrument. Instruments were usually developed by the authors, often with little mention of the test development criteria. The type of knowledge assessed was rarely specified. Most instruments included clinical vignettes. Instrument properties and psychometric characteristics were seldom reported.

CONCLUSION

There are a number of written assessments available in the literature. However, the reporting of the development and psychometric properties of these instruments is often incomplete. Practice implications written assessment of communication skills is widely used in health professions education. Improvement in the reporting of instrument development, items and psychometrics may help communication skills teachers better identify when, how and for whom written assessment of communication should be used.

Measuring hygiene competence: the picture-based situational judgement test HygiKo

BACKGROUND

With the onset of the COVID-19 pandemic at the beginning of 2020, the crucial role of hygiene in healthcare settings has once again become very clear. For diagnostic and for didactic purposes, standardized and reliable tests suitable to assess the competencies involved in "working hygienically" are required. However, existing tests usually use self-report questionnaires, which are suboptimal for this purpose. In the present study, we introduce the newly developed, competence-oriented HygiKo test instrument focusing health-care professionals' hygiene competence and report empirical evidence regarding its psychometric properties.

METHODS

HygiKo is a Situational Judgement Test (SJT) to assess hygiene competence. The HygiKo-test consists of twenty pictures (items), each item presents only one unambiguous hygiene lapse. For each item, test respondents are asked (1) whether they recognize a problem in the picture with respect to hygiene guidelines and, (2) if yes, to describe the problem in a short verbal response. Our sample comprised n = 149 health care professionals (79.1 % female; age: M = 26.7 years, SD = 7.3 years) working as clinicians or nurses. The written responses were rated by two independent raters with high agreement (α > 0.80), indicating high reliability of the measurement. We used Item Response Theory (IRT) for further data analysis.

RESULTS

We report IRT analyses that show that the HygiKo-test is suitable to assess hygiene competence and that it allows to distinguish between persons demonstrating different levels of ability for seventeen of the twenty items), especially for the range of low to medium person abilities. Hence, the HygiKo-SJT is suitable to get a reliable and competence-oriented measure for hygiene-competence.

CONCLUSIONS

In its present form, the HygiKo-test can be used to assess the hygiene competence of medical students, medical doctors, nurses and trainee nurses in cross-sectional measurements. In order to broaden the difficulty spectrum of the current test, additional test items with higher difficulty should be developed. The Situational Judgement Test designed to assess hygiene competence can be helpful in testing and teaching the ability of working hygienically. Further research for validity is needed.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

How can communicative competence instruction in medical studies be improved through digitalization?

The teaching of communicative competence plays an increasingly important role in medical education. In addition to traditional teaching formats, such as role-plays with simulated patients, technology-based approaches become more important in medical education. Teaching materials are increasingly augmented by videos of simulated doctor-patient conversations. This combination allows the content of teaching materials to be demonstrated with video or for videos to create a basis for reflection activities. In addition, conversation videos can illustrate different qualities of clinical communication and serve as illustrative material for describing particular issues in more detail. In addition to teaching clinical communicative competence, the assessment of this competence also plays an important role in medical educational research. So far, this has mainly been conducted through direct observation using checklists or rating scales. Relatively little is known about the assessment of communicative competence using standardized online-based tests. Situational Judgement Tests (SJTs) offer a promising approach in this respect. The BMBF-funded (BMBF = Bundesministerium für Bildung und Forschung - Federal Ministry of Education and Research) joint project voLeA (Entwicklung videobasierter Lehr- und Assessmentmodule zur Gesprächskompetenz im Medizinstudium = Development of video-based teaching and assessment modules for communicative competence in medical studies) addresses these two issues. Specifically, the project is engaged in developing e-learning modules to promote communicative competence and an assessment of this competence using an SJT. The present paper focuses on the benefits of technology-based learning and assessment units for clinical communicative competence in medical studies, using the voLeA project as an example.

Outcomes and outcome measures used in evaluation of communication training in oncology - a systematic literature review, an expert workshop, and recommendations for future research

BACKGROUND

Communication between health care provider and patients in oncology presents challenges. Communication skills training have been frequently developed to address those. Given the complexity of communication training, the choice of outcomes and outcome measures to assess its effectiveness is important. The aim of this paper is to 1) perform a systematic review on outcomes and outcome measures used in evaluations of communication training, 2) discuss specific challenges and 3) provide recommendations for the selection of outcomes in future studies.

METHODS

To identify studies and reviews reporting on the evaluation of communication training for health care professionals in oncology, we searched seven databases (Ovid MEDLINE, CENTRAL, CINAHL, EMBASE, PsychINFO, PsychARTICLES and Web of Science). We extracted outcomes assessed and the respective assessment methods. We held a two-day workshop with experts (n = 16) in communication theory, development and evaluation of generic or cancer-specific communication training and/or outcome measure development to identify and address challenges in the evaluation of communication training in oncology. After the workshop, participants contributed to the development of recommendations addressing those challenges.

RESULTS

Out of 2181 references, we included 96 publications (33 RCTs, 2 RCT protocols, 4 controlled trials, 36 uncontrolled studies, 21 reviews) in the review. Most frequently used outcomes were participants' training evaluation, their communication confidence, observed communication skills and patients' overall satisfaction and anxiety. Outcomes were assessed using questionnaires for participants (57.3%), patients (36.0%) and observations of real (34.7%) and simulated (30.7%) patient encounters. Outcomes and outcome measures varied widely across studies. Experts agreed that outcomes need to be precisely defined and linked with explicit learning objectives of the training. Furthermore, outcomes should be assessed as broadly as possible on different levels (health care professional, patient and interaction level).

CONCLUSIONS

Measuring the effects of training programmes aimed at improving health care professionals' communication skills presents considerable challenges. Outcomes as well as outcome measures differ widely across studies. We recommended to link outcome assessment to specific learning objectives and to assess outcomes as broadly as possible.

German Language Adaptation of the Kalamazoo Communication Skills Assessment Form (KCSAF): A Multi-Method Study of Two Cohorts of Medical Students

Abstract. This work investigates the German version of the Kalamazoo Communication Skills Assessment Form (KCSAFd) for three assessment methods: students' self-assessment (KCSAFd-self), assessment by standardised patients (KCSAFd-sPat) and video-assessment by trained raters (KCSAFd-video). Videotaped simulated patient consultations of N = 163 medical students from the first ( n = 97) and the final clinical years ( n = 66) were rated using the KCSAFd. Investigating the psychometric properties of the instrument, we found a two factor-construct with interpersonal and conversational competence. All methods showed good internal consistency and acceptable model fit values. Additionally, we found plausible relationships between the three methods and meaningful differences between the two groups of students.

Evaluation of a blended-learning training concept to train oncology physicians to advise their patients about complementary and integrative medicine (KOKON-KTO): study protocol for a prospective, multi-center, cluster-randomized trial

BACKGROUND

Many cancer patients are interested in complementary and integrative medicine during and after regular cancer treatment. Given the high number of users it is important that physicians and patients engage in a dialog about useful complementary and integrative medicine therapies during cancer treatment. In a prospective, multi-center, cluster-randomized evaluation study we will develop, implement and evaluate a training program for oncology physicians advising their patients on complementary and integrative medicine. The main objective of the study is to evaluate whether training physicians in a blended-learning approach (e-learning + skills-training workshop) in providing advice to their cancer patients on complementary and integrative medicine, in addition to handing out an information leaflet about reputable websites, has different effects on the outcomes of patients, physicians, and their interaction level, compared to only giving out the information leaflet.

METHODS/DESIGN

Forty-eight oncology physicians will be included into a cluster-randomized trial to either participate or not in the blended-learning training. Physicians will then advise 10 cancer patients each, resulting in 480 patients participating in the trial. The blended learning consists of nine units of up to 45 min of e-learning and 18 units of up to 45 min of on-site skills-training workshop focusing. Outcomes will be measured on the physician, patient, and physician-patient-interaction level.

DISCUSSION

A blended-learning program for oncology physicians to advise their cancer patients in a systematic way and a reasonable time frame on complementary and integrative medicine will be evaluated in depth in a large cluster-randomized trial.

TRIAL REGISTRATION

German Clinical Trials Register, ID: DRKS00012704 . Registered on 28 August 2017.

Development of a Web-Based Formative Self-Assessment Tool for Physicians to Practice Breaking Bad News (BRADNET)

BACKGROUND

Although most physicians in medical settings have to deliver bad news, the skills of delivering bad news to patients have been given insufficient attention. Delivering bad news is a complex communication task that includes verbal and nonverbal skills, the ability to recognize and respond to patients' emotions and the importance of considering the patient's environment such as culture and social status. How bad news is delivered can have consequences that may affect patients, sometimes over the long term.

OBJECTIVE

This project aimed to develop a Web-based formative self-assessment tool for physicians to practice delivering bad news to minimize the deleterious effects of poor way of breaking bad news about a disease, whatever the disease.

METHODS

BReaking bAD NEws Tool (BRADNET) items were developed by reviewing existing protocols and recommendations for delivering bad news. We also examined instruments for assessing patient-physician communications and conducted semistructured interviews with patients and physicians. From this step, we selected specific themes and then pooled these themes before consensus was achieved on a good practices communication framework list. Items were then created from this list. To ensure that physicians found BRADNET acceptable, understandable, and relevant to their patients' condition, the tool was refined by a working group of clinicians familiar with delivering bad news. The think-aloud approach was used to explore the impact of the items and messages and why and how these messages could change physicians' relations with patients or how to deliver bad news. Finally, formative self-assessment sessions were constructed according to a double perspective of progression: a chronological progression of the disclosure of the bad news and the growing difficulty of items (difficulty concerning the expected level of self-reflection).

RESULTS

The good practices communication framework list comprised 70 specific issues related to breaking bad news pooled into 8 main domains: opening, preparing for the delivery of bad news, communication techniques, consultation content, attention, physician emotional management, shared decision making, and the relationship between the physician and the medical team. After constructing the items from this list, the items were extensively refined to make them more useful to the target audience, and one item was added. BRADNET contains 71 items, each including a question, response options, and a corresponding message, which were divided into 8 domains and assessed with 12 self-assessment sessions. The BRADNET Web-based platform was developed according to the cognitive load theory and the cognitive theory of multimedia learning.

CONCLUSIONS

The objective of this Web-based assessment tool was to create a "space" for reflection. It contained items leading to self-reflection and messages that introduced recommended communication behaviors. Our approach was innovative as it provided an inexpensive distance-learning self-assessment tool that was manageable and less time-consuming for physicians with often overwhelming schedules.