Validating a script concordance test for assessing neurological localization and emergencies

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

PURPOSE

In the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT).

METHODS

The scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots.

RESULTS

Forty-five students completed the SCT. Their scores differed according to panel composition: 65.6 ± 9.73/100 points for nephrologists, and 70.27 ± 8.82 for the mixed panel, p < 0.001. Concordance between the scores was low with a bias of -4.27 ± 2.19 and a 95% limit of agreement of -8.96 to -0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 ± 2.6 places).

CONCLUSION

The composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.

Adapting to a Pandemic: Web-Based Residency Training and Script Concordance Testing in Emergency Medicine During COVID-19

OBJECTIVE

The coronavirus disease (COVID-19) pandemic necessitated alternative methods to ensure the continuity of medical education. Our study explores the efficacy and acceptability of a digital continuous medical education initiative for medical residents during this challenging period.

METHODS

From September to December 2020, 47 out of 60 enrolled trainee doctors participated in this innovative digital Continuous Medical Education (CME) approach. We utilized the Script Concordance Test to bolster clinical reasoning skills. Three simulation scenarios, namely Advanced Trauma Life Support (ATLS), Advanced Life Support (ALS), and European Paediatric Life Support (EPLS), were transformed into interactive online sessions via Zoom™. Participant feedback was also collected through a survey.

RESULTS

Consistent Script Concordance Testing (SCT) scores among participants indicated the effectiveness of the online training module. Feedback suggested a broad acceptance of this novel training approach. However, discrepancies observed between formative SCT scores, and summative Multiple-Choice Questions (MCQ) assessments highlighted areas for potential refinement.

CONCLUSIONS

Our findings showcase the resilience and adaptability of medical education amidst challenges like the global pandemic. The success of methodologies such as SCT, endorsed by prestigious bodies like the European Resuscitation Council and the American Heart Association, suggests their potential in preparing health care professionals for emergent situations. This research offers valuable insights for shaping future online CME strategies.

A scoping review of clinical reasoning research with Asian healthcare professionals

Clinical reasoning is the thought process that guides practice. Although a plethora of clinical reasoning studies in healthcare professionals exists, the majority appear to originate from Western cultures. A scoping review was undertaken to examine clinical reasoning related research across Asian cultures. PubMed, SciVerse Scopus, Web of Science and Airiti Library databases were searched. Inclusion criteria included full-text articles published in Asian countries (2007 to 2019). Search terms included clinical reasoning, thinking process, differential diagnosis, decision making, problem-based learning, critical thinking, healthcare profession, institution, medical students and nursing students. After applying exclusion criteria, n = 240 were included in the review. The number of publications increased in 2012 (from 5%, n = 13 in 2011 to 9%, n = 22) with a steady increase onwards to 12% (n = 29) in 2016. South Korea published the most articles (19%, n = 46) followed by Iran (17%, n = 41). Nurse Education Today published 11% of the articles (n = 26), followed by BMC Medical Education (5%, n = 13). Nursing and Medical students account for the largest population groups studied. Analysis of the articles resulted in seven themes: Evaluation of existing courses (30%, n = 73) being the most frequently identified theme. Only seven comparative articles showed cultural implications, but none provided direct evidence of the impact of culture on clinical reasoning. We illuminate the potential necessity of further research in clinical reasoning, specifically with a focus on how clinical reasoning is affected by national culture. A better understanding of current clinical reasoning research in Asian cultures may assist curricula developers in establishing a culturally appropriate learning environment.

Differential effects of team-based learning on clinical reasoning

BACKGROUND

Clinical reasoning (CR) is the ability to integrate information, knowledge and contextual factors for patient care. Few studies have explored effects of team-based learning (TBL) on neurological CR. This study compared simplified TBL (sTBL) against interactive lectures (IL) for teaching CR in neuroanatomical localisation (NL) and neurological emergencies (NE), assessed using a validated Script Concordance Test (SCT).

METHODS

A crossover study was conducted with third- and fifth-year undergraduates, randomly assigned to two groups, from the Yong Loo Lin School of Medicine in Singapore. Group 1 was taught NE with sTBL and NL with IL, whereas Group 2 was taught NL with sTBL and NE with IL. Teaching was conducted sequentially over 3 h followed immediately by the SCT. The primary outcome was the difference in mean SCT scores of NE and NL taught with sTBL versus IL.

FINDINGS

A total of 179 students (Group 1, n = 81; Group 2, n = 98) participated. Mean NL SCT scores for students taught with sTBL were significantly higher compared with IL (64.8% vs. 61.7%, mean difference 3.1%, 95% confidence interval [CI] 0.6%-5.5%, p = 0.013); effect size was 0.38. Mean NE SCT scores were similar between students taught with sTBL or IL (66.6% vs. 67.0%, mean difference -0.4%, 95% CI -2.2% to 3.1%, p = 0.75).

CONCLUSIONS

sTBL was superior to IL for teaching NL, whereas both methods were comparable for teaching NE. TBL may be suitable for teaching more complex neurological topics involving diagnostic reasoning through development of problem representation, hypothesis generation and illness script selection.

Pediatric Hospitalists' Performance and Perceptions of Script Concordance Testing for Self-Assessment

OBJECTIVES

The cognitive expertise of Pediatric Hospitalists (PH) lies not in standard knowledge but in making decisions under conditions of uncertainty. To maintain expertise, PH should engage in deliberate practice via self-assessments that promote higher-level cognitive processes necessary to address problems with missing or ambiguous information. Higher levels of cognition are purported with Script Concordance Test (SCT) questions compared to Multiple Choice Questions (MCQ). To determine if PH use higher levels of cognition when answering SCT versus MCQ questions and to analyze participants' perceptions of the utility of using SCT self-assessment for deliberate practice in addressing clinical problems encountered in daily practice.

METHODS

This is a mixed methods study comparing the cognitive level expressed according to Bloom's Taxonomy by PH answering MCQ versus SCT questions using a "think aloud" (TA) exercise, followed by qualitative analysis of interviews conducted afterward.

RESULTS

A significantly greater percentage of comments were coded as higher cognitive processes (apply, analyze, evaluate, and create) for SCT versus MCQ (74% vs 19%) compared with lower order (remember, understand); chi-square P < .00001. Analysis of interviews revealed 6 themes.

CONCLUSION

SCT questions elicited higher level cognition essential to clinical reasoning compared to MCQ questions. PH-indicated MCQ questions measure standard knowledge, while SCT questions better measure decision-making under conditions of uncertainty. PH-perceived SCT could be useful for deliberate practice in Pediatric Hospital Medicine decision-making if they could compare their rationale in answering questions with that of experts.

Evaluation of the use of script concordance test in a multicampus psychiatric pharmacy elective course

Introduction

Evaluating a student's ability to accept complexity, uncertainty, and ambiguity as part of clinical practice is difficult in a classroom setting using written tests. This study was conducted to explore the feasibility and validation of using a script concordance test (SCT) to evaluate pharmacy student knowledge and clinical competence in a psychiatry elective course.

Methods

This study involved prospective validation of psychiatry-focused SCT questions using a panel of practicing psychiatric pharmacists and retrospective review of student performance on the same SCT questions. The reliability of the SCT was also evaluated using Cronbach alpha coefficient.

Results

A total of 13 practicing psychiatric pharmacists participated in the validation phase of the study of 75 questions. Pharmacy student scores (n = 17) averaged 39.79 (±5.02) points, and psychiatric pharmacist scores averaged 50.11 (±4.51) points, representing mean percentages of 61.2% and 77.1%, respectively, on the adjusted exam. The Cronbach alpha was 0.94.

Discussion

The development of a valid and reliable SCT to test student psychiatric pharmacy knowledge and clinical competence after taking a psychiatry elective course was feasible.

Accuracy of script concordance tests in fourth-year medical students

OBJECTIVES

This investigation aimed to determine the validity of script concordance test (SCT), compared with clinical-case-related short-answer management problems (SAMP), in fourth-year medical students.

METHODS

This retrospective study was conducted at the Medical School of Lille University. Cardiology and gynecology examinations both included 3 SCT and 2 clinical-case-related SAMP.  Final score did not include SCT results, and was out of 20 points. The passing score was ≥10/20. Wilcoxon and McNemar tests were used to compare quantitative and qualitative variables, respectively. Correlation between scores was also analyzed.

RESULTS

A total of 519 and 521 students completed SAMP and SCT in cardiology and gynecology, respectively. Cardiology score was significantly higher in SCT than SAMP (mean ± SD 13.5±2.4 versus 11.4±2.6, Wilcoxon test, p<0.001). In gynecology, SCT score was significantly lower than SAMP score (10.8±2.6 versus 11.4±2.7, Wilcoxon test, p=0.001). SCT and SAMP scores were significantly correlated (p <0.05, Pearson's correlation). However, percentage of students with SCT score ≥ 10/20 was similar among those who passed or failed cardiology (327 of 359 (91%) vs 146 of 160 (91%), χ²=0.004, df =1, p=0.952), or gynecology (274 of 379 (65%) vs 84 of 142 (59%), χ²=1.614, df=1, p=0.204) SAMP test. Cronbach alpha coefficient was 0.31 and 0.92 for all SCT and SAMP, respectively.

CONCLUSIONS

 Although significantly correlated, the scores obtained in SCT and SAMP were significantly different in fourth-year medical students. These findings suggest that SCT should not be used for summative purposes in fourth-year medical students.