Constructing a question bank based on script concordance approach as a novel assessment methodology in surgical education

Test for clinical reasoning evaluation in Speech-Language Pathology: content validity

PURPOSE

To validate the content of the Speech-Language Pathology Concordance Test called FonoTCS.

METHODS

This is a content validation study of the instrument. Five speech-language pathologists, all with doctoral degrees and teaching experience, averaging 24.8 years of professional practice, participated in the development of FonoTCS and reached a consensus during the process. Thirty questions and 120 items were created, covering seven areas of speech-language pathology expertise across three domains. For content validation, FonoTCS was electronically sent to 15 evaluators to respond to a questionnaire with five questions, rated on a five-point scale, regarding the criteria of clarity, ethics, and relevance of the questions. The Corrected Content Validity Coefficient was calculated for all statements to analyze the responses. Questions with agreement percentages equal to or less than 80% were revised.

RESULTS

Thirteen evaluators, all female, with an average age of 39.07 years, including eight with master's degrees and five with doctoral degrees, and an average clinical practice experience of 15.38 years, participated in the analysis. The average Corrected Content Validity Coefficient values for the clarity criterion were 0.93 and 0.95, for the relevance criterion 0.98 and 0.92, and for the ethics criterion 0.99. Two questions received scores of 0.78 and 0.80, both related to the audiology area in the assessment/diagnosis domain, specifically question 2 regarding the relevance criterion. These questions were reviewed and restructured by the judges.

CONCLUSION

FonoTCS is a valid instrument from a content perspective.

Assessing clinical reasoning in airway related cases among anesthesiology fellow residents using Script Concordance Test (SCT)

INTRODUCTION

Clinical reasoning is a core competency for physicians. In the field of anesthesia, many situations require residents to use their clinical reasoning to make quick and appropriate decisions such as during emergency airway cases. The Script Concordance Test (SCT) is a test developed in recent years and validated that objectively assess clinical reasoning ability. However, studies involving SCT to assess clinical reasoning in airway management is scarce.

AIM

To evaluate SCT in assessing clinical reasoning for airway management in anesthesiology residents.

METHOD

A cross-sectional study involving residents and anesthesiology consultants from the Department of Anesthesiology and Intensive Care, Faculty of Medicine Universitas Indonesia was conducted to complete SCT. A panel of five anesthesiology consultants with more than 15 years of work experience constructed 20 SCT vignettes based on prevalent airway cases in our center from the past 10 years. Each SCT has three nested questions, with a total of 60 questions, to be answered within 120 min.

RESULTS

The SCT of 20 case vignettes with three nested questions were tested on 99 residents from the junior, intermediate, and senior residents, compared to answers from the expert group consisting of ten anesthesiology consultants with more than 5 years of experience. There were significant differences in mean SCT scores in the junior, intermediate, senior and expert groups, 59.3 (46.1-72.8), 64.7 (39.9-74.9), 67.5 (50.6-78.3), and 79.6 (78.4-84.8); p < 0,001 consecutively. Cronbach Alpha 0.69 was obtained, indicating good reliability.

CONCLUSION

Our SCT was proven to be a valid and reliable test instrument to assess the clinical reasoning in airway management for anesthesiology residents. SCT was able to discriminate between groups of different clinical experiences and should be included to evaluate airway competencies in anesthesiology residents.

An Ontology-Driven Learning Assessment Using the Script Concordance Test

Assessing the level of domain-specific reasoning acquired by students is one of the major challenges in education particularly in medical education. Considering the importance of clinical reasoning in preclinical and clinical practice, it is necessary to evaluate students’ learning achievements accordingly. The traditional way of assessing clinical reasoning includes long-case exams, oral exams, and objective structured clinical examinations. However, the traditional assessment techniques are not enough to answer emerging requirements in the new reality due to limited scalability and difficulty for adoption in online education. In recent decades, the script concordance test (SCT) has emerged as a promising tool for assessment, particularly in medical education. The question is whether the usability of SCT could be raised to a level high enough to match the current education requirements by exploiting opportunities that new technologies provide, particularly semantic knowledge graphs (SCGs) and ontologies. In this paper, an ontology-driven learning assessment is proposed using a novel automated SCT generation platform. SCTonto ontology is adopted for knowledge representation in SCT question generation with the focus on using electronic health records data for medical education. Direct and indirect strategies for generating Likert-type scores of SCT are described in detail as well. The proposed automatic question generation was evaluated against the traditional manually created SCT, and the results showed that the time required for tests creation significantly reduced, which confirms significant scalability improvements with respect to traditional approaches.

Multicentered Study Evaluating Pharmacy Students’ Perception of Palliative Care and Clinical Reasoning Using Script Concordance Testing

Introduction:

As the role of the pharmacist on the transdisciplinary palliative care team grows, the need for adequate instruction on palliative care and clinical reasoning skills in schools of pharmacy grows accordingly.

Methods:

This study evaluates second- and third-year pharmacy students from 6 accredited schools of pharmacy that participated in surveys before and after the delivery of a didactic palliative care elective. The survey collected student demographics, perceptions of the importance of and student skill level in palliative care topics. The script concordance test (SCT) was used to assess clinical decision-making skills on patient cases. Student scores on the SCT were compared to those of a reference panel of experts.

Results:

A total of 89 students completed the pre-/postsurveys and were included in data analysis. There was no statistically significant difference in student perceived importance of palliative care skills before and after the elective. Students from all 6 institutions showed significant increase in confidence in their palliative care skills at the end of the course. There was also a significant improvement across all institutions in clinical reasoning skills in most of the SCT questions used to assess these skills.

Conclusions:

Students choosing an elective in palliative care likely do so because they already have an understanding of the importance of these topics in their future practice settings. Delivery of a palliative care elective in the pharmacy curriculum significantly increases both student confidence in their palliative care skills and their clinical reasoning skills in these areas.

Accuracy of script concordance tests in fourth-year medical students

OBJECTIVES

This investigation aimed to determine the validity of script concordance test (SCT), compared with clinical-case-related short-answer management problems (SAMP), in fourth-year medical students.

METHODS

This retrospective study was conducted at the Medical School of Lille University. Cardiology and gynecology examinations both included 3 SCT and 2 clinical-case-related SAMP.  Final score did not include SCT results, and was out of 20 points. The passing score was ≥10/20. Wilcoxon and McNemar tests were used to compare quantitative and qualitative variables, respectively. Correlation between scores was also analyzed.

RESULTS

A total of 519 and 521 students completed SAMP and SCT in cardiology and gynecology, respectively. Cardiology score was significantly higher in SCT than SAMP (mean ± SD 13.5±2.4 versus 11.4±2.6, Wilcoxon test, p<0.001). In gynecology, SCT score was significantly lower than SAMP score (10.8±2.6 versus 11.4±2.7, Wilcoxon test, p=0.001). SCT and SAMP scores were significantly correlated (p <0.05, Pearson's correlation). However, percentage of students with SCT score ≥ 10/20 was similar among those who passed or failed cardiology (327 of 359 (91%) vs 146 of 160 (91%), χ²=0.004, df =1, p=0.952), or gynecology (274 of 379 (65%) vs 84 of 142 (59%), χ²=1.614, df=1, p=0.204) SAMP test. Cronbach alpha coefficient was 0.31 and 0.92 for all SCT and SAMP, respectively.

CONCLUSIONS

 Although significantly correlated, the scores obtained in SCT and SAMP were significantly different in fourth-year medical students. These findings suggest that SCT should not be used for summative purposes in fourth-year medical students.

Script concordance test in medical schools in Brazil: possibilities and limitations

CONTEXT AND OBJECTIVE

Routine use of the script concordance test (SCT) is not common in Brazilian universities. This study aimed to analyze application of the SCT in the medical school of a Brazilian university.

DESIGN AND SETTING

Quantitative, analytical and descriptive study in the medical school of a Brazilian university.

METHODS

A total of 159/550 students participated. The test comprised ten clinical cases within internal medicine, with five items per case, rated on a five-point Likert scale. The test was scored in accordance with a marking key that had been validated by a reference panel.

RESULTS

In the pre-clinical and clinical phases, the mean scores were 51.6% and 63.4% of the maximum possible scores, respectively. Comparison of the means of the responses among all the years showed that there were significant differences in 40% of the items. The panel marked all the possible answers in five items, while in one item, all the panelists marked a single answer. Cronbach's alpha was 0.64. The results indicated that the more senior students performed better. Construction of an SCT with discriminative questions was not easy. The low reliability index may have occurred due to: a) problems with the construction of the questions; b) limitations of the reference panel; and/or c) the scoring key.

CONCLUSION

This instrument is very difficult to construct, apply and correct. These difficulties may make application of an SCT as an assessment method unfeasible in units with limited resources.