Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research

Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research

From dual process to a family of theories known collectively as situativity, both micro and macro theories of cognition inform our current understanding of clinical reasoning (CR) and error. CR is a complex process that occurs in a complex environment, and a nuanced, expansive, integrated model of these theories is necessary to fully understand how CR is performed in the present day and in the future. In this perspective, we present these individual theories along with figures and descriptive cases for purposes of comparison before exploring the implications of a transtheoretical model of these theories for teaching, assessment, and research in CR and error.

CRISP: An Inpatient Pediatric Curriculum for Family Medicine Residents Using Clinical Reasoning and Illness Scripts

Introduction

Clinical reasoning enables safe patient care and is an important competency in medical education but can be challenging to teach. Illness scripts facilitate clinical reasoning but have not been used to create pediatric curricula.

Methods

We created CRISP (Clinical Reasoning with Illness Scripts in Pediatrics), a curriculum comprising four 1-hour learning sessions that deliberately incorporated clinical reasoning concepts and illness scripts to organize how four common chief complaints were taught to family medicine residents on inpatient pediatric rotations. We performed a multisite curriculum evaluation project over 6 months with family medicine residents at four institutions to assess whether the use of clinical reasoning concepts to structure CRISP was feasible and acceptable for learners and instructors and whether the use of illness scripts increased knowledge of four common pediatric chief complaints.

Results

For all learning sessions, family medicine residents and pediatric hospitalists agreed that CRISP's format was preferable to traditional didactic lectures. Pre-/posttest scores showed statistically significant increases in family medicine resident knowledge (respiratory distress [n = 42]: pretest, 72%, posttest, 92%; abdominal pain [n = 44]: pretest, 82%, posttest, 96%; acute febrile limp [n = 44]: pretest, 68%, posttest, 81%; well-appearing febrile infant [n = 42]: pretest, 58%, posttest, 73%; ps < .05).

Discussion

By using clinical reasoning concepts and illness script comparison to structure a pediatric curriculum, CRISP represents a novel instructional approach that can be used by pediatric hospitalists to increase family medicine resident knowledge about diagnoses associated with common pediatric chief complaints.

Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

PURPOSE

In the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT).

METHODS

The scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots.

RESULTS

Forty-five students completed the SCT. Their scores differed according to panel composition: 65.6 ± 9.73/100 points for nephrologists, and 70.27 ± 8.82 for the mixed panel, p < 0.001. Concordance between the scores was low with a bias of -4.27 ± 2.19 and a 95% limit of agreement of -8.96 to -0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 ± 2.6 places).

CONCLUSION

The composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.

Reliability of a script agreement test for undergraduate speech-language therapy students

PURPOSE

To estimate the reliability of scripts designed for undergraduate Speech-Language Therapy students.

METHODS

A descriptive cross-sectional study was carried out. Qualitative variables were summarized by frequency or proportion and quantitative through means (CI 95%). Reliability was estimated through Cronbach's α coefficient, and inter-rater agreement was determined using Fleiss's Kappa index. The analytical tests considered a significance level of p<0.05.

RESULTS

80 scripts organized in four areas of speech-language therapy were validated by 41 speech-language pathologists. The average experience of the professionals was 17.1 years. The reliability of the corpus was α: 0.67 (min= 0.34; max: 0.84), and the inter-rater agreement κ: 0.29 (min: 0.07; max: 0.45).

CONCLUSION

The corpus's reliability scores were similar to those reported by previous studies in different health professions. Having validated strategies aimed at developing proficiency and supporting classic training actions in undergraduate courses will contribute to increasing the quality of future health professionals.

Impact of training in the supervision of clinical reasoning in the pediatric emergency department on residents' perception of the on-call experience

BACKGROUND

In 2021, the prevalence of signs of burnout among medical residents was reported to be 67%. Being on call is particularly stressful for residents, notably due to their lack of medical experience. When they are on call, several factors contribute to a mismatch between the residents' theoretical knowledge and the operationalization of that knowledge in a clinical reasoning process. Using the script and cognitive load theories as a basis, we hypothesized that training clinician-teachers in the supervision of clinical reasoning could improve residents' perception of the experience of being on call.

METHODS

We performed a longitudinal, exploratory, controlled study with a cohort of medical residents who were on call in the pediatric emergency department during the semester from 1 November 2021 to 30 April 2022. During the night, the residents on call in the pediatric emergency department completed validated questionnaires investigating (1) mental effort, (2) cognitive weariness, (3) state anxiety, (4) feeling of self-efficacy, and (5) well-being. We compared the questionnaires of residents supervised by pediatricians trained in the supervision of clinical reasoning (supervision group) with those of residents in a control group, supervised by pediatricians with no specific pedagogical training.

RESULTS

A total of 284 questionnaires (174 supervision group, 110 controls) were collected from 38 residents in three pediatric emergency departments. The results confirm that being on call is difficult for residents. Compared to the control group, residents in the supervision group had lower cognitive weariness scores (mean 3.0 ± 1.1 vs. 3.5 ± 1.3). There was no significant difference between groups for any of the other dimensions of the on-call experience. In the supervision group, mental effort was significantly lower at the end of the study semester (5 [5-6] when on call in month 6 of the semester vs. 6 [5-7] when on call in months 1-5 of the semester; p = 0.01) and was greater for more senior residents (7 [6-8] for those in the 4th or higher semester of residency vs. 6 [5-7] for residents in their 1st, 2nd, or 3rd semester of residency; β = 0.92 ± 0.40; p = 0.02).

CONCLUSION

Beyond the positive effects for residents, this study illustrates the feasibility of implementing training for clinicians in the supervision of clinical reasoning.

Development and Validation of a Formative Assessment Tool for Nephrology Fellows' Clinical Reasoning

BACKGROUND

Diagnostic errors are commonly driven by failures in clinical reasoning. Deficits in clinical reasoning are common among graduate medical learners, including nephrology fellows. We created and validated an instrument to assess clinical reasoning in a national cohort of nephrology fellows and established performance thresholds for remedial coaching.

METHODS

Experts in nephrology education and clinical reasoning remediation designed an instrument to measure clinical reasoning through a written patient encounter note from a web-based, simulated AKI consult. The instrument measured clinical reasoning in three domains: problem representation, differential diagnosis with justification, and diagnostic plan with justification. Inter-rater reliability was established in a pilot cohort ( n =7 raters) of first-year nephrology fellows using a two-way random effects agreement intraclass correlation coefficient model. The instrument was then administered to a larger cohort of first-year fellows to establish performance standards for coaching using the Hofstee method ( n =6 raters).

RESULTS

In the pilot cohort, there were 15 fellows from four training program, and in the study cohort, there were 61 fellows from 20 training programs. The intraclass correlation coefficients for problem representation, differential diagnosis, and diagnostic plan were 0.90, 0.70, and 0.50, respectively. Passing thresholds (% total points) in problem representation, differential diagnosis, and diagnostic plan were 59%, 57%, and 62%, respectively. Fifty-nine percent ( n =36) met the threshold for remedial coaching in at least one domain.

CONCLUSIONS

We provide validity evidence for a simulated AKI consult for formative assessment of clinical reasoning in nephrology fellows. Most fellows met criteria for coaching in at least one of three reasoning domains, demonstrating a need for learner assessment and instruction in clinical reasoning.

Framework and Schema are False Synonyms: Defining Terms to Improve Learning

Clinical reasoning is an essential expertise of health care professionals that includes the complex cognitive processes that lead to diagnosis and management decisions. In order to optimally teach, learn, and assess clinical reasoning, it is imperative for teachers and learners to have a shared understanding of the language. Currently, educators use the terms schema and framework interchangeably but they are distinct concepts. In this paper, we offer definitions for schema and framework and use the high-stakes field of aviation to demonstrate the interplay of these concepts. We offer examples of framework and schema in the medical education field and discuss how a clear understanding of these concepts allows for greater intentionality when teaching and assessing clinical reasoning.

Implementation of three knowledge-oriented instructional strategies to teach clinical reasoning: Self-explanation, a concept mapping exercise, and deliberate reflection: AMEE Guide No. 150

The teaching of clinical reasoning is essential in medical education. This guide has been written to provide educators with practical advice on the design, development, and implementation of three knowledge-oriented instructional strategies for the teaching of clinical reasoning to medical students: Self-explanation (SE), a Clinical Reasoning Mapping Exercise (CREsME), and Deliberate Reflection (DR). We first synthesize the theoretical tenets that support the use of these strategies, including knowledge organization, and development of illness scripts. We then provide a detailed description of the key components of each strategy, emphasizing the practical applications of each one by sharing specific examples. We also explore the potential for a combined application of these strategies in a longitudinal and developmental approach to teaching clinical reasoning at the undergraduate level. Finally, we discuss enablers and barriers in the implementation and integration of these teaching strategies while taking into consideration curricular needs, context, and resources. We are aware that many strategies exist and are not arguing that SE, CReSME, and DR are the most effective ones or the only ones to be adopted. Nevertheless, we selected these strategies because of overarching theoretical principles, empirical evidence supporting their use, and our own experience with them. We are hoping to provide practical advice on the implementation of these strategies to practicing educators who aim at developing an integrated approach to the teaching of clinical reasoning to medical students at different stages of their development.

Developing Script Concordance Test Items in Otolaryngology to Improve Clinical Reasoning Skills: Validation using Consensus Analysis and Psychometrics

Background

Script concordance testing is widely practiced to foster and assess clinical reasoning. Our study aimed to develop script concordance test (SCT) in the specialty of otolaryngology and test the validation using panel response pattern and consensus index.

Materials and Methods

The methodology was an evolving pattern of constructing SCTs, administering them to the panel members, and optimizing the panel with response patterns and consensus index. The SCT's final items were chosen to be administered to the students.

Results

We developed 98 items of SCT and administered them to 20 panel members. The mean score of the panel members for these 98 items was 79.5 (standard deviation [SD] = 4.4). The consensus index calculated for the 98-item SCT ranged from 25.81 to 100. Sixteen items had bimodal and uniform response patterns; the consensus index improved when eliminated. We administered the rest 82 items of SCT to 30 undergraduate and ten postgraduate students. The mean score of undergraduate students was 61.1 (SD = 7.5) and that of postgraduate students was 67.7 (SD = 6.3). Cronbach's alpha for the 82-item SCT was 0.74. Excluding the 22 poor items, the final SCT instrument of 60 items had a Cronbach's alpha of 0.82.

Conclusion

Our study revealed that a consensus index above 60 had a good item-total correlation and be used to optimize the items for panel responses in SCT, necessitating further studies on this aspect. Our study also revealed that the panel response clustering pattern could be used to categorize the items, although bimodal and uniform distribution patterns need further differentiation.

Reasoning like a doctor or like a nurse? A systematic integrative review

When physicians and nurses are looking at the same patient, they may not see the same picture. If assuming that the clinical reasoning of both professions is alike and ignoring possible differences, aspects essential for care can be overlooked. Understanding the multifaceted concept of clinical reasoning of both professions may provide insight into the nature and purpose of their practices and benefit patient care, education and research. We aimed to identify, compare and contrast the documented features of clinical reasoning of physicians and nurses through the lens of layered analysis and to conduct a simultaneous concept analysis. The protocol of this systematic integrative review was published doi: 10.1136/bmjopen-2021-049862. A comprehensive search was performed in four databases (PubMed, CINAHL, Psychinfo, and Web of Science) from 30th March 2020 to 27th May 2020. A total of 69 Empirical and theoretical journal articles about clinical reasoning of practitioners were included: 27 nursing, 37 medical, and five combining both perspectives. Two reviewers screened the identified papers for eligibility and assessed the quality of the methodologically diverse articles. We used an onion model, based on three layers: Philosophy, Principles, and Techniques to extract and organize the data. Commonalities and differences were identified on professional paradigms, theories, intentions, content, antecedents, attributes, outcomes, and contextual factors. The detected philosophical differences were located on a care-cure and subjective-objective continuum. We observed four principle contrasts: a broad or narrow focus, consideration of the patient as such or of the patient and his relatives, hypotheses to explain or to understand, and argumentation based on causality or association. In the technical layer a difference in the professional concepts of diagnosis and the degree of patient involvement in the reasoning process were perceived. Clinical reasoning can be analysed by breaking it down into layers, and the onion model resulted in detailed features. Subsequently insight was obtained in the differences between nursing and medical reasoning. The origin of these differences is in the philosophical layer (professional paradigms, intentions). This review can be used as a first step toward gaining a better understanding and collaboration in patient care, education and research across the nursing and medical professions.

Does it make sense to use written instruments to assess communication skills? Systematic review on the concurrent and predictive value of written assessment for performance

OBJECTIVES

To evaluate possible associations between learners' results in written and performance-based assessments of communication skills (CS), either in concurrent or predictive study designs.

METHODS

Search included four databases for peer-reviewed studies containing both written and performance-based CS assessment. Eleven studies met the inclusion criteria.

RESULTS

Included studies predominantly assessed undergraduate medical students. Studies reported mainly low to medium correlations between written and performance-based assessment results (Objective Structured Clinical Examinations or encounters with simulated patients), and gave correlation coefficients ranging from 0.13 to 0.53 (p < 0.05). Higher correlations were reported when specific CS, like motivational interviewing were assessed. Only a few studies gave sufficient reliability indicators of both assessment formats.

CONCLUSIONS

Written assessment scores seem to predict performance-based assessments to a limited extent but cannot replace them entirely. Reporting of assessment instruments' psychometric properties is essential to improve the interpretation of future findings and could possibly affect their predictive validity for performance.

PRACTICE IMPLICATIONS

Within longitudinal CS assessment programs, triangulation of assessment including written assessment is recommended, taking into consideration possible limitations. Written assessments with feedback can help students and trainers to elaborate on procedural knowledge as a strong support for the acquisition and transfer of CS to different contexts.

Prueba de concordancia de guiones para entrenar el razonamiento clínico en estudiantes de fonoaudiología

La prueba de concordancia de guiones (PCG) ha sido utilizada en el entrenamiento y evaluación del razonamiento clínico (RC) como una estrategia innovadora en la formación de profesionales. Sin embargo, no se dispone de evidencia de su aplicación en el pregrado de fonoaudiología. El objetivo de esta investigación fue analizar el desempeño y la percepción de estudiantes de fonoaudiología con respecto al uso de scripts. Se diseñó un piloto pre-experimental y multicéntrico, complementado con tres grupos focales. Las variables cuantitativas continuas fueron resumidas a través de medias y desviación estándar. La comparación entre grupos se ejecutó con Anova one way y la prueba post hoc de Bonferroni, considerando un nivel de significancia p<.05. La fase cualitativa incorporó un análisis de contenido mediante la codificación abierta de textos y la identificación e interpretación de familias de significado emergentes. El rendimiento promedio de los estudiantes fue de 4.03 (DS= 0.35), observándose un incremento en el rendimiento de RC durante el semestre (p= 0.03). La percepción de los estudiantes resulto positiva y se identificó cuatro familias de significado relacionadas con: razonamiento clínico, oportunidades de mejora implementación de la estrategia y retroalimentación docente. A modo de conclusión, la incorporación de scripts en estudiantes de pregrado de fonoaudiología es factible, incrementa el rendimiento y apoya el desarrollo del RC.

Educating for adaptive expertise: case examples along the medical education continuum

Adaptive expertise represents the combination of both efficient problem-solving for clinical encounters with known solutions, as well as the ability to learn and innovate when faced with a novel challenge. Fostering adaptive expertise requires careful approaches to instructional design to emphasize deeper, more effortful learning. These teaching strategies are time-intensive, effortful, and challenging to implement in health professions education curricula. The authors are educators whose missions encompass the medical education continuum, from undergraduate through to organizational learning. Each has grappled with how to promote adaptive expertise development in their context. They describe themes drawn from educational experiences at these various learner levels to illustrate strategies that may be used to cultivate adaptive expertise.At Vanderbilt University School of Medicine, a restructuring of the medical school curriculum provided multiple opportunities to use specific curricular strategies to foster adaptive expertise development. The advantage for students in terms of future learning had to be rationalized against assessments that are more short-term in nature. In a consortium of emergency medicine residency programs, a diversity of instructional approaches was deployed to foster adaptive expertise within complex clinical learning environments. Here the value of adaptive expertise approaches must be balanced with the efficiency imperative in clinical care. At Mayo Clinic, an existing continuous professional development program was used to orient the entire organization towards an adaptive expertise mindset, with each individual making a contribution to the shift.The different contexts illustrate both the flexibility of the adaptive expertise conceptualization and the need to customize the educational approach to the developmental stage of the learner. In particular, an important benefit of teaching to adaptive expertise is the opportunity to influence individual professional identity formation to ensure that clinicians of the future value deeper, more effortful learning strategies throughout their careers.

Development and validation of a script concordance test to assess biosciences clinical reasoning skills: A cross-sectional study of 1st year undergraduate nursing students

BACKGROUND

Developing evaluative measures that assess clinical reasoning remains a major challenge for nursing education. A thorough understanding of biosciences underpins much of nursing practice and is essential to allow nurses to reason effectively. A gap in clinical reasoning can lead to unintended harm. The Script Concordance test holds promise as a measure of clinical reasoning in the context of uncertainty, situations common in nursing practice. The aim of this study is to develop and validate a test for first year undergraduate nursing students that will evaluate how bioscience knowledge is used to clinically reason.

METHODS

An international team, teaching biosciences to undergraduate nurses constructed a test integrating common clinical cases with a series of related test items: diagnostic, investigative and treatment. An expert panel (n = 10) took the test and commented on authenticity/ambiguities/omissions etc. This step is crucial for validity and for scoring of the student test. The test was administered to 47 first year undergraduate nursing students from the author sites. Students rated educational aspects of the tool both quantitatively and qualitatively. Statistical and content analyses inform the findings.

FINDINGS

Results indicate that the test is reliable and valid, differentiating between experts and students. Students demonstrated an ability to identify relevant data, link this to their bioscience content and predict outcomes (mean score = 50.78 ± 8.89). However, they lacked confidence in their answers when the scenarios appeared incomplete to them.

CONCLUSION

Nursing practice is dependent on a thorough understanding of biosciences and the ability to clinically reason. Script concordance tests can be used to promote both competencies. This method of evaluation goes further than probing factual knowledge. It also explores capacities of data interpretation, critical analysis, and clinical reasoning. Evaluating bioscience knowledge and real-world situations encountered in practice is a unique strength of this test.

The Use of Movement Scripts for Clinical Reasoning in Physical Therapist Education and Practice

The purpose of this Perspective is to present an application of script-based reasoning to physical therapist education and practice. Illness script-based reasoning has been described as a cognitive strategy for medical practitioners to diagnose and manage health conditions. Analogous to this medical model of patient management, "movement scripts" can be used by physical therapists in clinical reasoning. Movement scripts use features of the human movement system to recognize, categorize, and substantiate clinical problems and can be used to facilitate the development of master adaptive learners across the spectrum of physical therapist education and practice. Movement scripts are also consistent with the concept of the "human body as teacher" as the signature pedagogy proposed by the National Study of Excellence and Innovation in Physical Therapy Education. Movement of the human body, as captured by the concept of the human movement system, is a vehicle for lifelong adaptive learning for the physical therapist. Script-based learning and practice are consistent with other elements of this model, including practice-based learning and the creation of adaptive expertise. As the role of the movement system as a guide to physical therapist practice continues to evolve, movement scripts can provide a structure to facilitate development of clinical reasoning skills for physical therapist practice and education.

Clinical judgement by primary care physicians for the diagnosis of all-cause dementia or cognitive impairment in symptomatic people

BACKGROUND

In primary care, general practitioners (GPs) unavoidably reach a clinical judgement about a patient as part of their encounter with patients, and so clinical judgement can be an important part of the diagnostic evaluation. Typically clinical decision making about what to do next for a patient incorporates clinical judgement about the diagnosis with severity of symptoms and patient factors, such as their ideas and expectations for treatment. When evaluating patients for dementia, many GPs report using their own judgement to evaluate cognition, using information that is immediately available at the point of care, to decide whether someone has or does not have dementia, rather than more formal tests.

OBJECTIVES

To determine the diagnostic accuracy of GPs' clinical judgement for diagnosing cognitive impairment and dementia in symptomatic people presenting to primary care. To investigate the heterogeneity of test accuracy in the included studies.

SEARCH METHODS

We searched MEDLINE (Ovid SP), Embase (Ovid SP), PsycINFO (Ovid SP), Web of Science Core Collection (ISI Web of Science), and LILACs (BIREME) on 16 September 2021.

SELECTION CRITERIA

We selected cross-sectional and cohort studies from primary care where clinical judgement was determined by a GP either prospectively (after consulting with a patient who has presented to a specific encounter with the doctor) or retrospectively (based on knowledge of the patient and review of the medical notes, but not relating to a specific encounter with the patient). The target conditions were dementia and cognitive impairment (mild cognitive impairment and dementia) and we included studies with any appropriate reference standard such as the Diagnostic and Statistical Manual of Mental Disorders (DSM), International Classification of Diseases (ICD), aetiological definitions, or expert clinical diagnosis.

DATA COLLECTION AND ANALYSIS

Two review authors screened titles and abstracts for relevant articles and extracted data separately with differences resolved by consensus discussion. We used QUADAS-2 to evaluate the risk of bias and concerns about applicability in each study using anchoring statements. We performed meta-analysis using the bivariate method.

MAIN RESULTS

We identified 18,202 potentially relevant articles, of which 12,427 remained after de-duplication. We assessed 57 full-text articles and extracted data on 11 studies (17 papers), of which 10 studies had quantitative data. We included eight studies in the meta-analysis for the target condition dementia and four studies for the target condition cognitive impairment. Most studies were at low risk of bias as assessed with the QUADAS-2 tool, except for the flow and timing domain where four studies were at high risk of bias, and the reference standard domain where two studies were at high risk of bias. Most studies had low concern about applicability to the review question in all QUADAS-2 domains. Average age ranged from 73 years to 83 years (weighted average 77 years). The percentage of female participants in studies ranged from 47% to 100%. The percentage of people with a final diagnosis of dementia was between 2% and 56% across studies (a weighted average of 21%). For the target condition dementia, in individual studies sensitivity ranged from 34% to 91% and specificity ranged from 58% to 99%. In the meta-analysis for dementia as the target condition, in eight studies in which a total of 826 of 2790 participants had dementia, the summary diagnostic accuracy of clinical judgement of general practitioners was sensitivity 58% (95% confidence interval (CI) 43% to 72%), specificity 89% (95% CI 79% to 95%), positive likelihood ratio 5.3 (95% CI 2.4 to 8.2), and negative likelihood ratio 0.47 (95% CI 0.33 to 0.61). For the target condition cognitive impairment, in individual studies sensitivity ranged from 58% to 97% and specificity ranged from 40% to 88%. The summary diagnostic accuracy of clinical judgement of general practitioners in four studies in which a total of 594 of 1497 participants had cognitive impairment was sensitivity 84% (95% CI 60% to 95%), specificity 73% (95% CI 50% to 88%), positive likelihood ratio 3.1 (95% CI 1.4 to 4.7), and negative likelihood ratio 0.23 (95% CI 0.06 to 0.40). It was impossible to draw firm conclusions in the analysis of heterogeneity because there were small numbers of studies. For specificity we found the data were compatible with studies that used ICD-10, or applied retrospective judgement, had higher reported specificity compared to studies with DSM definitions or using prospective judgement. In contrast for sensitivity, we found studies that used a prospective index test may have had higher sensitivity than studies that used a retrospective index test.

AUTHORS' CONCLUSIONS

Clinical judgement of GPs is more specific than sensitive for the diagnosis of dementia. It would be necessary to use additional tests to confirm the diagnosis for either target condition, or to confirm the absence of the target conditions, but clinical judgement may inform the choice of further testing. Many people who a GP judges as having dementia will have the condition. People with false negative diagnoses are likely to have less severe disease and some could be identified by using more formal testing in people who GPs judge as not having dementia. Some false positives may require similar practical support to those with dementia, but some - such as some people with depression - may suffer delayed intervention for an alternative treatable pathology.

Script concordance test acceptability and utility for assessing medical students' clinical reasoning: a user's survey and an institutional prospective evaluation of students' scores

Script Concordance Testing (SCT) is a method for clinical reasoning assessment in the field of health-care training. Our aim was to assess SCT acceptability and utility with a survey and an institutional prospective evaluation of students' scores.With a user's online survey, we collected the opinions and satisfaction data of all graduate students and teachers involved in the SCT setting. We performed a prospective analysis comparing the scores obtained with SCT to those obtained with the national standard evaluation modality.General opinions about SCT were mostly negative. Students tended to express more negative opinions and perceptions. There was a lower proportion of negative responses in the teachers' satisfaction survey. The proportion of neutral responses was higher for teachers. There was a higher proportion of positive positions towards all questions among teachers. PCC scores significantly increased each year, but SCT scores increased only between the first and second tests. PCC scores were found significantly higher than SCT scores for the second and third tests. Medical students' and teachers' global opinion on SCT was negative. At the beginning SCT scores were found quite similar to PCC scores. There was a higher progression for PCC scores through time.

The Student-Generated Reasoning Tool (SGRT): Linking medical knowledge and clinical reasoning in preclinical education

INTRODUCTION

The simultaneous integration of knowledge acquisition and development of clinical reasoning in preclinical medical education remains a challenge. To help address this challenge, the authors developed and implemented the Student-Generated Reasoning Tool (SGRT)-a tool asking students to propose and justify pathophysiological hypotheses, generate findings, and critically appraise information.

METHODS

In 2019, students in a first-year preclinical course (n = 171; SGRT group) were assigned to one of 20 teams. Students used the SGRT individually, then in teams, and faculty provided feedback. The control group (n = 168) consisted of students from 2018 who did not use SGRT. Outcomes included academic performance, effectiveness of collaborative environments using the SGRT, and student feedback.

RESULTS

Students were five times more likely to get questions correct if they were in the SGRT group versus control group. Accuracy of pathophysiological hypotheses was significantly lower for individuals than teams. Qualitative analysis indicated students benefited from generating their own data, justifying their reasoning, and working individually as well as in teams.

CONCLUSIONS

This study introduces the SGRT as a potentially engaging, case-based, and collaborative learning method that may help preclinical medical students become aware of their knowledge gaps and integrate their knowledge in basic and clinical sciences in the context of clinical reasoning.

Evaluating the Clinical Reasoning of Student Health Professionals in Placement and Simulation Settings: A Systematic Review

(1) Background: Clinical reasoning is essential to the effective practice of autonomous health professionals and is, therefore, an essential capability to develop as students. This review aimed to systematically identify the tools available to health professional educators to evaluate students' attainment of clinical reasoning capabilities in clinical placement and simulation settings. (2) Methods: A systemic review of seven databases was undertaken. Peer-reviewed, English-language publications reporting studies that developed or tested relevant tools were included. Searches included multiple terms related to clinical reasoning and health disciplines. Data regarding each tool's conceptual basis and evaluated constructs were systematically extracted and analysed. (3) Results: Most of the 61 included papers evaluated students in medical and nursing disciplines, and over half reported on the Script Concordance Test or Lasater Clinical Judgement Rubric. A number of conceptual frameworks were referenced, though many papers did not reference any framework. (4) Conclusions: Overall, key outcomes highlighted an emphasis on diagnostic reasoning, as opposed to management reasoning. Tools were predominantly aligned with individual health disciplines and with limited cross-referencing within the field. Future research into clinical reasoning evaluation tools should build on and refer to existing approaches and consider contributions across professional disciplinary divides.

How Clinicians Perceive Artificial Intelligence-Assisted Technologies in Diagnostic Decision Making: Mixed Methods Approach

Background

With the rapid development of artificial intelligence (AI) and related technologies, AI algorithms are being embedded into various health information technologies that assist clinicians in clinical decision making.

Objective

This study aimed to explore how clinicians perceive AI assistance in diagnostic decision making and suggest the paths forward for AI-human teaming for clinical decision making in health care.

Methods

This study used a mixed methods approach, utilizing hierarchical linear modeling and sentiment analysis through natural language understanding techniques.

Results

A total of 114 clinicians participated in online simulation surveys in 2020 and 2021. These clinicians studied family medicine and used AI algorithms to aid in patient diagnosis. Their overall sentiment toward AI-assisted diagnosis was positive and comparable with diagnoses made without the assistance of AI. However, AI-guided decision making was not congruent with the way clinicians typically made decisions in diagnosing illnesses. In a quantitative survey, clinicians reported perceiving current AI assistance as not likely to enhance diagnostic capability and negatively influenced their overall performance (β=–0.421, P=.02). Instead, clinicians’ diagnostic capabilities tended to be associated with well-known parameters, such as education, age, and daily habit of technology use on social media platforms.

Conclusions

This study elucidated clinicians’ current perceptions and sentiments toward AI-enabled diagnosis. Although the sentiment was positive, the current form of AI assistance may not be linked with efficient decision making, as AI algorithms are not well aligned with subjective human reasoning in clinical diagnosis. Developers and policy makers in health could gather behavioral data from clinicians in various disciplines to help align AI algorithms with the unique subjective patterns of reasoning that humans employ in clinical diagnosis.

Educators and practitioners' perspectives in the development of a learning by concordance tool for medical clerkship in the context of the COVID pandemic

BACKGROUND

The COVID-19 pandemic has forced medical schools to create educational material to palliate the anticipated and observed decrease in clinical experiences during clerkships. An online learning by concordance (LbC) tool was developed to overcome the limitation of students' experiences with clinical cases. However, knowledge about the instructional design of an LbC tool is scarce, especially the perspectives of collaborators involved in its design: 1- educators who wrote the vignettes' questions and 2- practitioners who constitute the reference panel by answering the LbC questions. The aim of this study was to describe the key elements that supported the pedagogical design of an LbC tool from the perspectives of educators and practitioners.

METHODS

A descriptive qualitative research design has been used. Online questionnaires were used, and descriptive analysis was conducted.

RESULTS

Six educators and 19 practitioners participated in the study. Important to the educators in designing the LbC tool were prevalent or high-stake situations, theoretical knowledge, professional situations experienced and perceived difficulties among students, and that the previous workshop promoted peer discussion and helped solidify the writing process. Important for practitioners was standards of practice and consensus among experts. However, they were uncertain of the educational value of their feedback, considering the ambiguity of the situations included in the LbC tool.

CONCLUSIONS

The LbC tool is a relatively new training tool in medical education. Further research is needed to refine our understanding of the design of such a tool and ensure its content validity to meet the pedagogical objectives of the clerkship.

Script concordance tests: A call for action in dental education

The Script Concordance Test (SCT) is an educational tool that aims to assess the ability to interpret medical information under conditions of uncertainty. It is widely used and validated in health education, but almost unknown in dentistry. Based on authentic clinical problem-solving situations, it allows to assess clinical reasoning that experienced health workers develop over the years. A specific scoring system, dedicated to SCT, considers the variability of responses of practitioners in the same clinical situations. Finally, the scores generated by SCT reflect the respondents' ability to interpret clinical data compared to experienced clinicians. This article aims to familiarise the dental educators' community with SCT construction, optimisation and its possible applications.

Relationship between web-based illness scripts and the performance of medical students in orthopedic surgery placements

OBJECTIVES

We aimed to explore the relationship between web-based orthopedic illness scripts and medical students' performance as assessed through examination results.

METHODS

This was a retrospective cohort study with 83 fifth-year medical students in an academic hospital. During a one-month placement, they were instructed to do web-based illness scripts. Their performances were assessed by examination in the last week. All recorded data about illness scripts and examination results were retrieved. The students were separated into high and low response groups based on completed illness scripts. The characteristics of the students between the two groups were compared. Pearson correlation coefficients and regression analysis were used to identify the relationship between illness scripts and examination results.

RESULTS

There were 56 students in the high-response and 27 in the low-response groups. The characteristics and examination scores were not significantly different between the groups, while there was a significant difference in script completion (t₍₂₇₎=13.72, p<0.001). Using Pearson correlation, we found weak correlations without significance between completed scripts, illness script scores, and examination scores. We found no relationship between illness script scores and examination scores, even in the high response group, by regression analysis.

CONCLUSIONS

The use of web-based orthopedic illness scripts did not correlate to the examination performance of medical students. A high number of scripts without variety and limited time for practicing may have obscured potential positive relationships. Illness scripts should be adjusted as appropriate for each school before being assigned. A further multi-center, prospective study is suggested to identify the correlations and investigate the influencing factors.

Script Concordance Approach in Nursing Education

BACKGROUND

The script concordance approach aims at triggering judgments in simulated contexts of uncertainty.

PROBLEM

Nursing students need to be prepared to manage the uncertainty of clinical practice.

APPROACH

The purpose of this article is to describe the theoretical foundation and the pedagogical use of the script concordance approach, as well as to present the current state of nursing evidence on the subject. The script concordance approach includes (1) script concordance testing, which is a quantitative examination that evaluates clinical reasoning; (2) a face-to-face script concordance activity; and (3) a digital educational strategy based on script concordance delivered via an online teaching/learning platform that aims to support clinical reasoning development.

CONCLUSIONS

Relying on questioning and experts' modeling, the script concordance offers an innovative pedagogical approach that approximates the uncertainty of clinical practice.

Learning by concordance (LbC) to develop professional reasoning skills: AMEE Guide No. 141

Developing effective clinical reasoning is central to health professions education. Learning by concordance (LbC) is an on-line educational strategy that makes learners practice reasoning competency in case-based clinical situations. The questions asked are similar to those professionals ask themselves in their practice and participant answers are compared to those of a reference panel. When participants answer the questions, they receive an automated feedback that is two-fold as they see (1) how the panelists respond and (2) justifications each panelist gives for their answer. This provides rich contextual knowledge about the situation, supplemented by a synthesis summarizing crucial points. As many educators in the health sciences are engaging in introducing innovative approaches, many consider building LbC learning modules. Elaborating, designing and implementing a LbC tool remain a challenge. This AMEE Guide describes the steps and elements to be considered when designing a LbC tool, drawing on examples from distinct health professions: medicine, nursing, physiotherapy, and dentistry. Specifically, the following elements will be discussed: (1) LbC theoretical underpinnings; (2) principles of LbC questioning; (3) goals of the concordance-based activity; (4) nature of reasoning tasks; (5) content/levels of complexity; (6) reference panel; (7) feedback/synthesis messages; (8) on-line learning platforms.

Diagnostic reasoning: relationships among expertise, accuracy, and ways that nurse practitioner students self-explain

OBJECTIVES

To improve diagnostic ability, educators should employ multifocal strategies. One promising strategy is self-explanation, the purposeful technique of generating self-directed explanations during problem-solving. Students self-explain information in ways that range from simple restatements to multidimensional thoughts. Successful problem-solvers frequently use specific, high-quality self-explanation types. In a previous phase of research, unique ways that family nurse practitioner (NP) students self-explain during diagnostic reasoning were identified and described. This study aims to (a) explore relationships between ways of self-explaining and diagnostic accuracy levels and (b) compare differences between students of varying expertise in terms of ways of self-explaining and diagnostic accuracy levels. Identifying high-quality diagnostic reasoning self-explanation types may facilitate development of more refined self-explanation educational strategies.

METHODS

Thirty-seven family NP students enrolled in the Doctor of Nursing Practice program at a large, Midwestern university diagnosed three written case studies while self-explaining. During the quantitative phase of a content analysis, associational and comparative data analysis techniques were applied.

RESULTS

Expert students voiced significantly more clinical and biological inference self-explanations than did novice students. Diagnostic accuracy scores were significantly associated with biological inference scores. Clinical and biological inference scores accounted for 27% of the variance in diagnostic accuracy scores, with biological inference scores significantly influencing diagnostic accuracy scores.

CONCLUSIONS

Not only were biologically focused self-explanations associated with diagnostic accuracy, but also their spoken frequency influenced levels of diagnostic accuracy. Educational curricula should support students to view patient presentations in terms of underlying biology from the onset of their education.

Unravelling the polyphony in clinical reasoning research in medical education

RATIONALE

Clinical reasoning lies at the heart of medical practice and has a long research tradition. Nevertheless, research is scattered across diverse academic disciplines with different research traditions in a wide range of scientific journals. This polyphony is a source of conceptual confusion.

AIMS AND OBJECTIVES

We sought to explore the underlying theoretical assumptions of clinical reasoning aiming to promote a comprehensive conceptual and theoretical understanding of the subject area. In particular, we asked how clinical reasoning is defined and researched and what conceptualizations are relevant to such uses.

METHODS

A scoping review of the clinical reasoning literature was undertaken. Using a "snowball" search strategy, the wider scientific literature on clinical reasoning was reviewed in order to clarify the different underlying conceptual assumptions underlying research in clinical reasoning, particularly to the field of medical education. This literature included both medical education, as well as reasoning research in other academic disciplines outside medical education, that is relevant to clinical reasoning. A total of 124 publications were included in the review.

RESULTS

A detailed account of the research traditions in clinical reasoning research is presented. In reviewing this research, we identified three main conceptualisations of clinical reasoning: "reasoning as cognitive activity," "reasoning as contextually situated activity," and "reasoning as socially mediated activity." These conceptualisations reflected different theoretical understandings of clinical reasoning. Each conceptualisation was defined by its own set of epistemological assumptions, which we have identified and described.

CONCLUSIONS

Our work seeks to bring into awareness implicit assumptions of the ongoing clinical reasoning research and to hopefully open much needed channels of communication between the different research communities involved in clinical reasoning research in the field.

Diagnostic reasoning in internal medicine: a practical reappraisal

The practice of clinical medicine needs to be a very flexible discipline which can adapt promptly to continuously changing surrounding events. Despite the huge advances and progress made in recent decades, clinical reasoning to achieve an accurate diagnosis still seems to be the most appropriate and distinctive feature of clinical medicine. This is particularly evident in internal medicine where diagnostic boundaries are often blurred. Making a diagnosis is a multi-stage process which requires proper data collection, the formulation of an illness script and testing of the diagnostic hypothesis. To make sense of a number of variables, physicians may follow an analytical or an intuitive approach to clinical reasoning, depending on their personal experience and level of professionalism. Intuitive thinking is more typical of experienced physicians, but is not devoid of shortcomings. Particularly, the high risk of biases must be counteracted by de-biasing techniques, which require constant critical thinking. In this review, we discuss critically the current knowledge regarding diagnostic reasoning from an internal medicine perspective.

How do cognitive processes influence script concordance test responses?

INTRODUCTION

The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses.

METHODS

Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview.

RESULTS

Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility.

CONCLUSION

The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

L’apprentissage du raisonnement clinique infirmier dans le cadre d’un dispositif éducatif numérique basé sur la concordance de scripts

Contexte : Un des défis actuels des responsables de la formation des futurs infirmiers est de concevoir des stratégies éducatives pour favoriser le développement optimal du raisonnement clinique infirmier (RCI).But : L’étude visait à développer, mettre à l’essai et évaluer un dispositif éducatif numérique basé sur la concordance de scripts auprès d’étudiants en sciences infirmières. Méthodes : Un devis de recherche-développement a permis de soutenir les étapes de réalisation du dispositif éducatif. Une validation de contenu a été intégrée au devis à l’aide d’une technique Delphi modifiée. Les choix de réponses des étudiants et des experts aux questions du dispositif ont été compilés. Un questionnaire en ligne et des entrevues de groupe focalisées ont été utilisés auprès d’étudiants afin d’évaluer l’acceptabilité et l’utilisabilité du dispositif éducatif. Résultats : Cinq experts-collaborateurs ont validé le contenu du dispositif éducatif. La version finale comptait 81 questions liées à 22 vignettes. Douze experts-panélistes et quarante-cinq étudiants ont répondu aux questions du dispositif éducatif. Les indices alpha (α) de Cronbach ont oscillé entre 0,87 et 0,90. Globalement, les étudiants ont apprécié l’utilisation du dispositif éducatif, dont la rétroaction variée et formative d’experts à des situations apparentées à la vie professionnelle. Conclusion : Le dispositif éducatif repose sur l’utilisation d’un test de concordance de scripts numérisé auquel des rétroactions d’experts sont intégrées à des fins d’apprentissage du RCI. Essentiel, l’engagement des experts-panélistes inscrit la conception d’un tel dispositif éducatif dans une démarche de co-construction et de validation du matériel pédagogique.

Barriers to Learning Clinical Reasoning: a Qualitative Study of Medicine Clerkship Students

Clinical reasoning is crucial to good patient care, but both learning and applying clinical reasoning skills in the context of a complex working environment can be challenging. We sought to understand the perceived barriers to learning clinical reasoning, as experienced by internal medicine clerkship students at our institution. We invited internal medicine clerkship students to participate in focus groups to discuss their experiences with and barriers to learning clinical reasoning. A survey was administered to gather additional responses. Responses were reviewed, coded, and synthesized to identify key themes. Twenty-nine medicine clerkship students (male = 14, female = 15) participated in six 60-minute focus groups, and 121 (61% response rate) students responded to the barriers to clinical reasoning survey from March 2018 to May 2019. We identified three themes (clerkship acclimation, data access, and practice optimization) and ten subthemes as aspects of the clerkship environment that impacted students' ability to develop clinical reasoning skills. Students identified barriers to learning clinical reasoning during the internal medicine clerkship. The themes "clerkship acclimation" and "data access" were identified as prerequisites to clinical reasoning while the theme "practice optimization" described key components of the deliberate practice of clinical reasoning. Educators and health systems may improve the development of clinical reasoning by recognizing and overcoming these barriers within clinical learning environments.

Learning strategies used by undergraduate nursing students in the context of a digitial educational strategy based on script concordance: A descriptive study

BACKGROUND

the digital educational strategy based on script concordance is an educational method that has been attracting increasing attention in healthcare education programs to fostering the development of clinical reasoning. It includes a digitized Script Concordance Test with incorporated expert feedback. However, the learning strategies required of students in the context of its use remain unknown.

OBJECTIVE

This study aimed to identify the learning strategies that undergraduate nursing students need to use in the context of the digital educational strategy based on script concordance.

METHOD

A qualitative descriptive design was used to identify student learning strategies. Data was collected using an online questionnaire and semi-directed focus group interviews. Bégin's taxonomy provided the framework for linking the data collected to learning strategies required of students.

RESULTS

Forty-four students participated in the study. Results show that when using a digital educational strategy based on script concordance, students are called to rely on their nascent scripts in order to select the data in short ill-defined clinical vignettes, evaluate new information repeatedly, anticipate microjudgments, and thus, gradually increase their knowledge and refine their scripts. Viewing the experts' feedback and consulting the referencing tools helped students self-monitor their knowledge, a key metacognitive strategy to learning clinical reasoning. Completed individually or with peers, the digital educational strategy could be used to learn a particular concept or as an integrative activity before an evaluation.

CONCLUSION

This original study has allowed us to link nursing clinical reasoning teaching conditions to the learning strategies used to develop this competency. Study results inform instructors about digital educational strategy based on script concordance to make it complementary with other educational strategies to better support complex learning of nursing clinical reasoning.

Simulated Mass Casualty Incident Triage Exercise for Training Medical Personnel

Audience

The target audience is any medical professional who requires training in mass casualty incident (MCI) triage. This could apply to pre-hospital specialists, nurses, medical students, residents, and physicians.

Introduction

Emergency medicine specialists must be able to triage patients quickly, especially in an MCI scenario. The simple triage and rapid treatment (START) system allows providers to categorize patients according to the urgency with which patients must access limited resources. Providers should be comfortable utilizing the START triage system before an MCI or disaster so that they can be prepared to implement it if necessary. This exercise uses simulation and gamification as instructional strategies to encourage knowledge of and comfort with the START triage system for emergency providers.

Educational Objectives

By the end of this exercise, learners should be able to (1) recite the basic START patient categories (2) discuss the physical exam signs associated with each START category, (3) assign roles to medical providers in a mass casualty scenario, (4) accurately categorize patients into triage categories: green, yellow, red, and black, and (5) manage limited resources when demand exceeds availability.

Educational Methods

Gamification is the use of elements of game design in non-game contexts.1 Gamification was implemented in this scenario by assigning participants to roles and teams, while creating an engaging, fun, and competitive environment. The exercise also uses low fidelity simulation (without simulation equipment) to encourage learners to practice using the START triage system in a low stakes environment.2 It is possible for the learners to be divided into two groups that each have the same patients, resources, and objectives. The team that finishes triaging all patients first would be declared the winner. However, in our implementation, we completed the exercise as a single group of learners and patients.

Research Methods

Learners were given a survey at the end of implementation and also given the opportunity to discuss feedback with the instructors in a group discussion after completing the exercise. There was no formal assessment completed after the exercise.

Results

Informal feedback was collected at the end of the exercise. Residents and medical students all enjoyed the experience. The feedback was overwhelmingly positive. All participants providing feedback stated they would enjoy participating in the exercise again and suggested that it is implemented annually for review of triage topics. We also received informal feedback for suggested changes which we will discuss in this article. An optional, anonymous survey was given to participants at the end of the exercise. There were six responses. Of those surveyed, 100% of participants stated the effectiveness and value of the exercise was outstanding (a rating of five on a scale of one to five). Regarding the quality of the exercise, and whether the participants felt engaged, 100% of responses gave a rating of five. When asked to consider the relevance of the session, 100% of participants selected a score of five ("I loved this session"). Regarding whether the content was applicable to practice of emergency medicine, 80% of respondents stated the session was highly relevant and 20% of responses selected a score of mostly relevant. One question asked for points of improvement for the session to which there were no responses.

Discussion

Learners were assigned roles in the exercise by the incident commander, fulfilling objective three. The START categories were discussed at the beginning of the exercise by the lead proctor (using PowerPoint) and then utilized throughout the exercise, thus accomplishing objectives one and two. The residents/students filling the triage roles were primarily responsible for fulfilling objective four; however, all participants assisted in categorization of patients throughout the exercise. Finally, objective five was addressed through the various social situations and complications that can be implemented during the exercise. We chose not to implement the additional "radiation contamination" scenario (details available in the article text) due to time constraints; however, this is an additional option to address objective five. The implementation was effective based on informal feedback from participants and proctors as well as evidenced by the responses to the anonymous survey. Learners found the aspects of resource management, review of START triage, repetition of the START triage system, and medical management of various types of trauma informative and meaningful. We received valuable feedback from both learners and proctors, which we will discuss in this article.

Topics

Mass casualty incident, disaster, START, gamification, simulation, emergency medicine, triage, triage category, contamination, teamwork, trauma, projectile trauma, penetrating injury, blunt trauma, intracranial hemorrhage, fracture, trauma in pregnancy, active shooter, radiation, radio communication.

Impact of panelists' experience on script concordance test scores of medical students

BACKGROUND

The evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.

METHODS

In 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.

RESULTS

A panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.

CONCLUSIONS

Even though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.

Widening the lens on teaching and assessing clinical reasoning: from "in the head" to "out in the world"

Traditional teaching and assessment of clinical reasoning has focused on the individual clinician because of the preeminence of the information processing (IP) theory perspective. The clinician's mind has been viewed as the main source of effective or ineffective reasoning, and other participants, the environment and their interactions have been largely ignored. A social cognitive theoretical lens could enhance our understanding of how reasoning and error and the environment are linked. Therefore, a new approach in which the clinical reasoning process is situated and examined within the context may be required. The theories of embodied cognition, ecological psychology, situated cognition (SitCog) and distributed cognition (DCog) offer new insights to help the teacher and assessor enhance the quality of clinical reasoning instruction and assessment. We describe the teaching and assessment implications of clinical reasoning and error through the lens of this family of theories. Direct observation in different contexts focused on individual and team performance, simulation (with or without enhancement of technology), stimulated recall, think-aloud, and modeling are examples of teaching and assessment strategies grounded in this family of social cognitive theories. Educators may consider the instructional design of learning environments and educational tools that promote a situated educational approach to the teaching and assessment of clinical reasoning.

The Management Script: A Practical Tool for Teaching Management Reasoning

Management reasoning, a component of clinical reasoning, has become an important area for medical education research given its inherent complexity, role in medical decision making, and association with high-value care. Teaching management reasoning requires characterizing its core concepts and identifying strategies to teach them. In this Perspective, the authors propose the term "management script" to describe the mental schema that clinicians develop and use in medical decision making. Management scripts are high-level, precompiled, conceptual knowledge structures of the courses of action that a clinician may undertake to address a patient's health care problem(s). Like illness scripts, management scripts have foundational elements that are shared by most clinicians but are ultimately idiosyncratic based on each clinician's unique history of learning and experience. Applying management scripts includes 2 steps-(1) management script activation and (2) management option selection-which can occur reflexively (unconsciously) or deliberately (consciously), similar to, respectively, the System 1 thinking and System 2 thinking of dual process theory. Management scripts can be taught for different conditions by using management script templates, educational scaffolds that provide possible courses of action to address a health care problem at any stage. Just as learners use system-based or organ-based frameworks to generate a differential diagnosis, students can use a generic management script template early in training to develop management scripts for specific problems. Future research directions include exploring the role of management scripts in medical education and quality improvement practices.

Making Decisions in the Era of the Clinical Decision Rule: How Emergency Physicians Use Clinical Decision Rules

PURPOSE

Physicians are often asked to integrate clinical decision rules (CDRs) with their own cognitive processes to reach a diagnosis. Clinicians, researchers, and educators must understand these cognitive processes to evaluate and improve the diagnostic process. The authors sought to explore emergency physicians' diagnostic processes and to examine how they integrated CDRs into their reasoning using simulated cases (with chest pain or leg pain).

METHOD

From August 2015 to July 2016, 16 practicing emergency physicians from 3 teaching hospitals associated with McMaster University, Ontario, Canada, were interviewed via a novel "teach aloud" protocol. Six videos of simulated patients with chest pain, breathlessness, or leg discomfort were used as prompts for the physicians to demonstrate their diagnostic thinking. Using a constructivist grounded theory analysis, 3 investigators independently reviewed the interview transcripts, meeting regularly to discuss identified themes and subthemes until sufficiency was reached.

RESULTS

A model to describe how clinicians integrate their own decision making with CDRs was developed, showing that physicians engage in an iterative diagnostic process that repeatedly refines the differential diagnosis list. The steps in the diagnostic process were: refinement of the differential diagnosis, ordering a hierarchy of risk, the decision to test, choosing the tests, and interpreting test results. Physicians applied CDRs when they had already decided to test.

CONCLUSIONS

To date, CDRs assume a static, linear model of clinical decision making. Findings demonstrate that participants engaged in iterative and dynamic decision-making processes that changed throughout their patient encounter, contingent on multiple contextual features. Understanding these processes could inform future development of CDRs and educational strategies around these decision aids.

Effects of reflection and immediate feedback to improve clinical reasoning of medical students in the assessment of dermatologic conditions: a randomised controlled trial

BACKGROUND

There are few studies that directly compared different interventions to improve medical students' clinical reasoning for dermatologic conditions.

OBJECTIVE

To investigate the effectiveness of adding practice with reflection and immediate feedback on traditional dermatology electives in improving medical students' ability in evaluating skin lesions.

METHODS

The participants were fourth-year medical students of Seoul National University College of Medicine, Korea, who were enrolled to take a 2-week dermatology elective course (n = 87). Students were assigned to one of the three educational interventions: 2-h training involving 10 written clinical cases (experimental); 1-h lecture and 1-h outpatient clinic (lecture); and 2-h outpatient clinic (no intervention). Before and at the end of rotation, diagnostic accuracy was estimated using 20 written clinical cases with photographs (10 novel cases presented in diagnostic training [training set], 10 cases with diagnoses not included in training [control set]).

RESULTS

There was a significant interaction effect of intervention×set×time. A post hoc analysis indicated that the students in the experimental group outperformed students in the other two groups only in the training set of the final tests; after completing the 2-week rotation, for the training set, the mean score was higher in the experimental group (7.5 ± 1.3) than in the lecture (5.7 ± 1.6) and no intervention (5.6 ± 1.3) groups, producing an effect size of 1.2 standard deviation (SD) and 1.5 SD, respectively.

CONCLUSION

Practicing written clinical cases with reflection and feedback is superior to a lecture-based approach and yields additional benefits to a dermatology elective, thereby enhancing medical students' ability to accurately diagnose skin lesions.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT03472001. Registered 21 March 2018.

Evaluation of an intervention to support the development of clinical problem solving skills during a hospital-based experiential learning program for South African pharmacy students

INTRODUCTION

Final year pharmacy students at the Nelson Mandela University described feeling unprepared for the complexities of clinical reasoning and problem solving during the hospital-based experiential learning program (ELP). The fifteen week ELP is part of the third Pharmacology and Therapeutics module taken in the final year of the four year Bachelor of Pharmacy (BPharm) degree program. An intervention, in the form of supplementary academic support sessions, was implemented during the ELP. The aim of the research was to evaluate the intervention from the perspectives of the students' experience as well as academic performance in the ELP.

METHODS

A mixed-methods approach using a quasi-experimental, intervention-based design was employed. Final year students in Year 1 (control cohort) completed the ELP as usual without additional academic support. In the consecutive academic year (Year 2), the intervention cohort completed the ELP that was enhanced with the intervention. Student feedback provided qualitative data, while quantitative data in the form of assessment marks were used as a measure of academic performance.

RESULTS

The students' experience of the intervention was overwhelming positive and provided evidence that the students felt better prepared for patient case analysis as a result of the intervention. A small but significant improvement in academic performance in the ELP was also seen.

CONCLUSIONS

The intervention, using a modified team based learning (TBL) approach developed from student-identified needs, was successfully implemented and provided a means of enhancing the development of problem solving and clinical reasoning skills during the ELP, through additional academic support sessions.

How to develop clinical reasoning in medical students and interns based on illness script theory: An experimental study

Background: Although theory explains the development of illness script, it does not provide answers how medical students develop scripts in their learning. To fill the knowledge gap of developing illness script in medical students and interns, this study aimed to investigate the impact of educational strategies inspired by theory in the development of illness scripts.

Methods: A total of 15 medical students and 12 interns participated in an educational intervention that included theory-driven strategies. To evaluate the impact of this intervention, clinical reasoning problem (CRP) and key features (KF) tests were used for before and after the intervention. In addition to descriptive statistics, the differences in participants’ pretest and posttest variables were tested using Wilcoxon. Significance level was set at p≤0.05 for all tests.

Results: Interns significantly recognized more KF in the posttest. However, no significant difference was found between the pretest and posttest scores in total diagnostic accuracy (5.41±1.16 vs 4.91±1.44; p=0.111) and total correct discriminating score (0.41±0.66 vs 1.41±2.06; p=0.146). Medical students produced less total key features in the posttest, indicating that they became less elaborate in their case processing. However, no significant difference was observed in common KF score (0.4 [0.25-0.78] vs 0.9 [0.6-1]; p=0.791) and discriminative key features score (0.33 [0.16-0.33] vs 0.22 [0.11-0.44]; p=0.972) in the posttest compared to the pretest.

Conclusion: This study showed that theory-driven educational strategies have an impact on illness script development specifically in interns. It is recommended that this intervention would be tested on those in higher levels of expertise (ie, residents).

The cognitive process of test takers when using the script concordance test rating scale

CONTEXT

Clinical decision making (CDM) skills are important to learn and assess in order to establish competence in trainees. A common tool for assessing CDM is the script concordance test (SCT), which asks test takers to indicate how a new clinical finding influences a proposed plan using a Likert-type scale. Most criticisms of the SCT relate to its rating scale but are largely theoretical. The cognitive process of test takers when selecting their responses using the SCT rating scale remains understudied, but is essential to gathering validity evidence for use of the SCT in CDM assessment.

METHODS

Cases from an SCT used in a national validation study were administered to 29 residents and 14 staff surgeons. Semi-structured cognitive interviews were then conducted with 10 residents and five staff surgeons based on the SCT results. Cognitive interview data were independently coded by two data analysts, who specifically sought to elucidate how participants mapped their internally generated responses to any of the rating scale options.

RESULTS

Five major issues were identified with the response matching cognitive process: (a) the meaning of the '0' response option; (b) which response corresponds to agreement with the planned management; (c) the rationale for picking '±1' versus '±2'; (d) which response indicates the desire to undertake the planned management plus an additional procedure, and (e) the influence of time on response selection.

CONCLUSIONS

Studying how test takers (experts and trainees) interpret the SCT rating scale has revealed several issues related to inconsistent and unintended use. Revising the scale to address the variety of interpretations could help to improve the response process validity of the SCT and therefore improve the SCT's ability to be used in CDM skills assessments.

Electronic charts do not facilitate the recognition of patient hazards by advanced medical students: A randomized controlled study

Chart review is an important tool to identify patient hazards. Most advanced medical students perform poorly during chart review but can learn how to identify patient hazards context-independently. Many hospitals have implemented electronic health records, which enhance patient safety but also pose challenges. We investigated whether electronic charts impair advanced medical students’ recognition of patient hazards compared with traditional paper charts. Fifth-year medical students were randomized into two equal groups. Both groups attended a lecture on patient hazards and a training session on handling electronic health records. One group reviewed an electronic chart with 12 standardized patient hazards and then reviewed another case in a paper chart; the other group reviewed the charts in reverse order. The two case scenarios (diabetes and gastrointestinal bleeding) were used as the first and second case equally often. After each case, the students were briefed about the patient safety hazards. In total, 78.5% of the students handed in their notes for evaluation. Two blinded raters independently assessed the number of patient hazards addressed in the students’ notes. For the diabetes case, the students identified a median of 4.0 hazards [25%–75% quantiles (Q25–Q75): 2.0–5.5] in the electronic chart and 5.0 hazards (Q25–Q75: 3.0–6.75) in the paper chart (equivalence testing, p = 0.005). For the gastrointestinal bleeding case, the students identified a median of 5.0 hazards (Q25–Q75: 4.0–6.0) in the electronic chart and 5.0 hazards (Q25–Q75: 3.0–6.0) in the paper chart (equivalence testing, p < 0.001). We detected no improvement between the first case [median 5.0 (Q25–Q75: 3.0–6.0)] and second case [median, 5.0 (Q25–Q75: 3.0–6.0); p < 0.001, test for equivalence]. Electronic charts do not seem to facilitate advanced medical students’ recognition of patient hazards during chart review and may impair expertise formation.

National Board of Medical Examiners and Curriculum Change: What Do Scores Tell Us? A Case Study at the University of Balamand Medical School

OBJECTIVES

This study describes the results of NBME (National Board of Medical Examiners) implementation in Balamand Medical School (BMS) from 2015 to 2019, after major curricular changes were introduced as of 2012. BMS students' performance was compared with the international USMLE step 1 (United States Medical Licensing Examination, herein referred to as step 1) cohorts' performances. The BMS students' NBME results were analyzed over the successive academic years to assess the impact of the serial curricular changes that were implemented.

METHODS

This longitudinal study describes the performance of BMS preclinical second year medicine (Med II) students on all their NBME exams over 4 academic years starting 2015-2016 to 2018-2019. These scores were compared with the step 1 comparison group scores using item difficulty. The t test was computed for each of the NBME exams to check whether the scores' differences were significant.

RESULTS

Results revealed that all BMS cohorts scored lower than the international USMLE step 1 comparison cohorts in all disciplines across the 4 academic years except Psychiatry. However, the results were progressively approaching step 1 results, and the difference between step 1 scores and BMS students' NBME scores became closer and not significant as of year 4.

CONCLUSIONS

The results of the study are promising. They show that the serial curricular changes enabled BMS Med II students' scores to reach the international cohorts' scores after 4 academic years. Moreover, the absence of statistical difference between cohort 4 scores and step 1 cohorts is not module dependent and applies to all clinical modules. Further studies should be conducted to assess whether the results obtained for cohort 4 can be maintained.

A Randomized, Crossover Pilot Study of a Novel Web-Based/Mobile Platform for Collaborative Small Group Practice in Therapeutic Reasoning

BACKGROUND AND PURPOSE

Therapeutic reasoning-the mental process of making judgments and decisions about treatment-is developed through acquisition of knowledge and application in actual or simulated experiences. Health professions education frequently uses collaborative small group work to practice therapeutic reasoning. This pilot study compared the impact of a web-based/mobile tool for collaborative case work and discussion to usual practice on student perceptions and performance on questions designed to test therapeutic knowledge and reasoning.

METHODS

In a therapeutics course that includes case-based workshops, student teams of 3 to 4 were randomly assigned to usual workshop preparation (group SOAP sheet) or preparation using the Practice Improvement using Virtual Online Training (PIVOT) platform. PIVOT was also used in the workshop to review the case and student responses. The next week, groups crossed over to the other condition. Students rated favorability with the preparatory and in-workshop experiences and provided comments about the PIVOT platform via a survey. Student performance on examination items related to the 2 workshop topics was compared.

RESULTS

One hundred and eleven students (94%) completed post-workshop surveys after both workshops. The majority of students (57%) preferred using the PIVOT platform for workshop collaboration. Favorability ratings for the in-workshop experience did not change significantly from first to second study week, regardless of sequence of exposure. There was no relationship between examination item scores and the workshop platform the students were exposed to for that content (P = .29). Student responses highlighted the efficiency of working independently before collaborating as a group and the ability to see other students' thought processes as valuable aspects of PIVOT. Students expressed frustration with the PIVOT user interface and the lack of anonymity when discussing their answers in the workshop.

CONCLUSION

A web-based/mobile platform for student team collaboration on therapeutic reasoning cases discussed in small group settings yielded favorable ratings, examination performance comparable to standard approaches, and was preferred by a majority of students. During the rapid shift to substantial online learning for the COVID-19 pandemic, virtual collaboration tools like PIVOT may help health professions teachers to better support groups working virtually on scaffolded therapeutic reasoning tasks.

The diagnostic expertise acceleration module (DEAM): promoting the formation of organized knowledge

Background: Ensuring that learners acquire diagnostic competence in a timely fashion is critical to providing high quality and safe patient care. Resident trainees typically gain experience by undertaking repetitive clinical encounters and receiving feedback from supervising faculty. By critically engaging with the diagnostic process, learners encapsulate medical knowledge into discrete memories that are able to be recollected and refined in subsequent clinical encounters. In the setting of exponentially increasing medical complexity and current duty hour limitations, the opportunities for successful practice in the clinical arena have become limited. Novel educational methods are needed to more efficiently bridge the gap from novice to expert diagnostician. Objective: Using a conceptual framework which incorporates deliberate practice, script theory, and learning curves, we developed an educational module prototype to coach novice learners to formulate organized knowledge (i.e. a repertoire of illness scripts) in an accelerated fashion thereby simulating the ideal experiential learning in a clinical rotation. Design: We developed the Diagnostic Expertise Acceleration Module (DEAM), a web-based module for learning illness scripts of diseases causing pediatric respiratory distress. For each case, the learner selects a diagnosis, receives structured feedback, and then creates an illness script with a subsequent expert script for comparison. Results: We validated the DEAM with seven experts, seven experienced learners and five novice learners. The module data generated meaningful learning curves of diagnostic accuracy. Case performance analysis and self-reported feedback demonstrated that the module improved a learner's ability to diagnose respiratory distress and create high-quality illness scripts. Conclusions: The DEAM allowed novice learners to engage in deliberate practice to diagnose clinical problems without a clinical encounter. The module generated learning curves to visually assess progress towards expertise. Learners acquired organized knowledge through formulation of a comprehensive list of illness scripts.

A Systems Approach to Design and Implementation of Patient Assessment Tools in the Inpatient Setting

Failure to rescue events, or events involving preventable deaths from complications, are a significant contributor to inpatient mortality. While many interventions have been designed and implemented over several decades, this patient safety issue remains at the forefront of concern for most hospitals. In the first part of this study, the development and implementation of one type of highly studied and widely adopted rescue intervention, algorithm-based patient assessment tools, is examined. The analysis summarizes how a lack of systems-oriented approaches in the design and implementation of these tools has resulted in suboptimal understanding of patient risk of mortality and complications and the early recognition of patient deterioration. The gaps identified impact several critical aspects of excellent patient care, including information-sharing across care settings, support for the development of shared mental models within care teams, and access to timely and accurate patient information. This chapter describes the use of several system-oriented design and implementation activities to establish design objectives, model clinical processes and workflows, and create an extensible information system model to maximize the benefits of patient state and risk assessment tools in the inpatient setting. A prototype based on the product of the design activities is discussed along with system-level considerations for implementation. This study also demonstrates the effectiveness and impact of applying systems design principles and practices to real-world clinical applications.

A protocol analysis of use of forward and backward reasoning during valuation problem solving

Purpose

Behavioural studies of valuers have suggested that valuers rely on a number of cognitive strategies involving reasoning and intuition when undertaking a valuation task. However, there are few studies of the actual reasoning mechanisms in valuation. In other fields, much attention has been paid to forward and backward reasoning, as this shows the choices and decisions that are made in undertaking a complex task. This paper studied this during a valuation task. The purpose of this paper is twofold: first, to develop a methodological approach for empirical research on valuers’ reasoning, and, second, to report expert-novice differences on valuers’ use of forward and backward reasoning during a valuation problem solving.

Design/methodology/approach

The study utilised a verbal protocol analysis (VPA) to elicit think-aloud data from a purposive sample of a group of valuers of different levels of expertise undertaking a commercial-valuation task. Through a content analysis interpretive strategy, the transcripts were analysed into different cognitive segments identifying the forward and backward reasoning strategies.

Findings

The findings showed that valuers accomplished the valuation task by dividing the overall problem into sub-problems. These sub-problems are thereafter solved by integrating available data with existing knowledge by relying more on forward reasoning than backward reasoning. However, there were effects associated with the level of expertise in the way the processes of forward and backward reasoning are used, with the expert and intermediate valuers being more thorough and comprehensive in their reasoning process than the novices.

Research limitations/implications

This study explores the possibility that forward and backward reasoning play an important role in commercial valuation problem solving using a limited sample of valuers. Given this, data cannot be generalised to all valuation practice settings but may motivate future research that examines the effectiveness of forward and backward reasoning in diverse valuation practice settings and develops a holistic model of valuation reasoning.

Practical implications

The findings of this study are applicable to valuation practice. Future training efforts need to evaluate the usefulness of teaching problem solving and explicitly recognise forward and backward reasoning, along with other problem-solving strategies uncovered in this study, as standard training strategies for influencing the quality of valuation decisions.

Originality/value

By adopting VPA, this study employs an insightful and rich dataset which allows an interpretation of thoughts of valuers into cognitive reasoning strategies that provide a deeper level of understanding of how valuers solve valuation problem; this has not been possible in previous related valuation studies.