Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test

Does it make sense to use written instruments to assess communication skills? Systematic review on the concurrent and predictive value of written assessment for performance

OBJECTIVES

To evaluate possible associations between learners' results in written and performance-based assessments of communication skills (CS), either in concurrent or predictive study designs.

METHODS

Search included four databases for peer-reviewed studies containing both written and performance-based CS assessment. Eleven studies met the inclusion criteria.

RESULTS

Included studies predominantly assessed undergraduate medical students. Studies reported mainly low to medium correlations between written and performance-based assessment results (Objective Structured Clinical Examinations or encounters with simulated patients), and gave correlation coefficients ranging from 0.13 to 0.53 (p < 0.05). Higher correlations were reported when specific CS, like motivational interviewing were assessed. Only a few studies gave sufficient reliability indicators of both assessment formats.

CONCLUSIONS

Written assessment scores seem to predict performance-based assessments to a limited extent but cannot replace them entirely. Reporting of assessment instruments' psychometric properties is essential to improve the interpretation of future findings and could possibly affect their predictive validity for performance.

PRACTICE IMPLICATIONS

Within longitudinal CS assessment programs, triangulation of assessment including written assessment is recommended, taking into consideration possible limitations. Written assessments with feedback can help students and trainers to elaborate on procedural knowledge as a strong support for the acquisition and transfer of CS to different contexts.

Prueba de concordancia de guiones para entrenar el razonamiento clínico en estudiantes de fonoaudiología

La prueba de concordancia de guiones (PCG) ha sido utilizada en el entrenamiento y evaluación del razonamiento clínico (RC) como una estrategia innovadora en la formación de profesionales. Sin embargo, no se dispone de evidencia de su aplicación en el pregrado de fonoaudiología. El objetivo de esta investigación fue analizar el desempeño y la percepción de estudiantes de fonoaudiología con respecto al uso de scripts. Se diseñó un piloto pre-experimental y multicéntrico, complementado con tres grupos focales. Las variables cuantitativas continuas fueron resumidas a través de medias y desviación estándar. La comparación entre grupos se ejecutó con Anova one way y la prueba post hoc de Bonferroni, considerando un nivel de significancia p<.05. La fase cualitativa incorporó un análisis de contenido mediante la codificación abierta de textos y la identificación e interpretación de familias de significado emergentes. El rendimiento promedio de los estudiantes fue de 4.03 (DS= 0.35), observándose un incremento en el rendimiento de RC durante el semestre (p= 0.03). La percepción de los estudiantes resulto positiva y se identificó cuatro familias de significado relacionadas con: razonamiento clínico, oportunidades de mejora implementación de la estrategia y retroalimentación docente. A modo de conclusión, la incorporación de scripts en estudiantes de pregrado de fonoaudiología es factible, incrementa el rendimiento y apoya el desarrollo del RC.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Challenges with international medical graduate selection: finding positive attributes predictive of success in family medicine residency

Background

Criteria to select residents most likely to succeed, other than proficiency of their medical knowledge, is a challenge facing preceptors. International Medical Graduates (IMGs) play an integral role in mitigating the high demand for family medicine physicians across Canada. Thus, selecting IMG candidates that have a high probability of succeeding in Canadian educational settings is important. The purpose of this study is to elucidate qualitative attributes that positively correspond to success in residency, to ultimately assist in the selection of IMG residents most likely to achieve family medicine residency.

Methods

Interviews of 13 family medicine preceptors from some of the largest IMG training sites in Canada were performed to collect original data. The data was coded in tandem sequences using standardized coding techniques to increase robustness of results.

Results

The identified positive predictors of an IMG residents’ success are: presence of a positive attitude, proficient communication skills, high level of clinical knowledge, trainability.

Conclusions

The results provide adequate guidelines to assist in selection of IMG residents. Canada is a unique sociocultural setting where standardized selection methods of IMGs have not been employed. By selecting IMG residents who possess these attributes upon inception of residency, benefits of instruction will be maximized and result in residents developing increased aptitudes for patient care.

Application of test-enhanced learning (TEL) in obstetrics and gynecology: a prospective study

OBJECTIVE

Clinical reasoning is an essential skill, the foundations of which should be acquired during medical school. Within the format of test-based learning, such examinations can also be used to support the long-term retention of procedural knowledge necessary for clinical reasoning. The aim was to investigate whether repeated exposure to clinical cases in obstetrics and gynecology (OBGYN) with built-in questions leads to higher learning outcome than pure reading cases and what influence the delay between the intervention and the final test has on the retention of the respective content.

METHODS

In this non-randomised crossover study, 5th-year medical students (duration of the study is 6 years) taking a 1-week clinical attachment in OBGYN participated in computer-based case seminars in winter term 2020/2021, in which different case histories on gynecological-obstetric diseases were presented. Case content was identical for all groups, but the presentation format (cases with key feature questions vs read-only cases) of individual case vignettes changed weekly. The also intervention was repeated after 2 weeks for each group. Knowledge was assessed in an entry and an exit exam consisting of 40 short-answer questions.

RESULTS

A total of 94 out of 118 eligible students participated in the study (response rate: 79.7%). Learning outcome was significantly higher for items presented in the key feature format compared to items presented as read-only cases (74.2 ± 8.6% vs. 71.0 ± 9.2%; p = 0.017). Furthermore, the analysis showed that the temporal distance of the intervention package from the final examination had no influence on retention.

CONCLUSION

This is the first study to demonstrate an effect of test-enhanced learning on clinical reasoning in the subject of OGBYN. In this cross-over study, repeated testing was more effective than repeated case-based learning alone. Curricular implementation of longitudinal key feature testing can thus improve learning outcomes for OBGYN.

Knowledge self-monitoring, efficiency, and determinants of self-confidence statement in multiple choice questions in medical students

BACKGROUND

Multiple-choice question (MCQ) tests are commonly used to evaluate medical students, but they do not assess self-confidence nor penalize lucky guess or harmful behaviors. Based on a scoring method according to the appropriateness of confidence in answers, the study aimed at assessing knowledge self-monitoring and efficiency, and the determinants of self-confidence.

METHODS

A cross-sectional study of 842 s- and third-year medical students who were asked to state their level of confidence (A: very confident, B: moderately confident and C: not confident) during 12 tests (106,806 events). A bonus was applied if the level of confidence matched with the correctness of the answer, and a penalty was applied in the case of inappropriate confidence.

RESULTS

Level A was selected more appropriately by the top 20% students whereas level C was selected more appropriately by the lower 20% students. Efficiency of higher-performing students was higher when correct (among correct answers, rate of A statement), but worse when incorrect compared to the bottom 20% students (among incorrect answers, rate of C statement). B and C statements were independently associated with female and male gender, respectively (OR for male vs female = 0.89 [0.82-0.96], p = 0.004, for level B and 1.15 [1.01-1.32], p = 0.047, for level C).

CONCLUSION

While both addressing the gender confidence gap, knowledge self-monitoring might improve awareness of students' knowledge whereas efficiency might evaluate appropriate behavior in clinical practice. These results suggest differential feedback during training in higher versus lower-performing students, and potentially harmful behavior in decision-making during clinical practice in higher-performing students.

Impact of panelists' experience on script concordance test scores of medical students

BACKGROUND

The evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.

METHODS

In 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.

RESULTS

A panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.

CONCLUSIONS

Even though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.

Physical therapists' perspectives of the construct of motor learning, and their motor learning-based practice: a qualitative study

Introduction: The perceptions and beliefs of physical therapists play an important role in their clinical behavior. These clinician-related aspects may contribute to the knowledge-action gap in the field of motor learning (ML). Accordingly, this study explored how a sample of physical therapists perceive ML in their practice.Methods: A qualitative phenomenological study was conducted. Twelve physical therapists with a wide range of work experience were purposefully selected. Data were collected through semi-structured interviews and analyzed using content analysis methodology. Themes and categories were identified.Results: We identified five key themes: 1) 'ML within the scope of PT' showed that participants perceived ML as a fundamental part of the profession; 2) 'Understanding of ML' showed that participants had difficulty understanding the field comprehensively; 3) 'Self-efficacy toward the topic of ML' revealed that participants had low self-efficacy about their knowledge of the topic; 4) 'Features of ML implementation' showed that physical therapists tended to implement ML intuitively; and 5) 'Barriers to implementation and ways to facilitate it' showed that lack of knowledge was perceived as a main barrier to implementation.Conclusions: Physical therapists' perceptions of and experiences with ML showed the complexity of the field and the lack of clarity regarding its theoretical content and clinical applications. There is a need to reinforce knowledge of ML among physical therapists at different levels of professional development and to promote self-efficacy toward its use, possibly by increasing ML knowledge and education while emphasizing its clinical context.

Assessing Clinical Reasoning: Targeting the Higher Levels of the Pyramid

Clinical reasoning is a core component of clinical competency that is used in all patient encounters from simple to complex presentations. It involves synthesis of myriad clinical and investigative data, to generate and prioritize an appropriate differential diagnosis and inform safe and targeted management plans.The literature is rich with proposed methods to teach this critical skill to trainees of all levels. Yet, ensuring that reasoning ability is appropriately assessed across the spectrum of knowledge acquisition to workplace-based clinical performance can be challenging.In this perspective, we first introduce the concepts of illness scripts and dual-process theory that describe the roles of analytic system 1 and non-analytic system 2 reasoning in clinical decision making. Thereafter, we draw upon existing evidence and expert opinion to review a range of methods that allow for effective assessment of clinical reasoning, contextualized within Miller's pyramid of learner assessment. Key assessment strategies that allow teachers to evaluate their learners' clinical reasoning ability are described from the level of knowledge acquisition, through to real-world demonstration in the clinical workplace.

[Analysis of the discrimination of the final marks after the first computerized national ranking exam in Medicine in June 2016 in France]

INTRODUCTION

The first computerised national ranking exam (cNRE) in Medicine was introduced in June 2016 for 8214 students. It was made of 18 progressive clinical cases (PCCs) with multiple choice questions (MCQs), 120 independent MCQs and 2 scientific articles to criticize. A lack of mark discrimination grounded the cNRE reform. We aimed to assess the discrimination of the final marks after this first cNRE.

METHODS

A national Excel^® file gathering overall statistics and marks were transmitted to the medical faculties after the cNRE. The mean points deviation between two papers and the percentage of points ranking 75% of students allowed us to analyse marks' discrimination.

RESULTS

The national distribution sigmoid curve of the marks is superimposable with previous NRE in 2015. In PCCs, 72% of students were ranked in 1090 points out of 7560 (14%). In independents MCQs, 73% of students were ranked in 434 points out of 2160 (20%). In critical analysis of articles, 75% of students were ranked in 225 points out of 1080 (21%). The above percentages of students are on the plateau of each discrimination curve for PCCs, independent MCQs and critical analysis of scientific articles.

CONCLUSION

The cNRE reduced equally-ranked students compared to 2015, with a mean deviation between two papers of 0.28 in 2016 vs 0.04 in 2015. Despite the new format introduced by the cNRE, 75% of students are still ranked in a low proportion of points that is equivalent to previous NRE in 2015 (between 15 et 20% of points).

Standardized Assessment of Reasoning in Contexts of Uncertainty

Current written tools of assessment are mostly measuring the capacity to solve well-defined problems by the application of rules and principles, while the essence of expertise in the professions lies in the capacity to solve illdefined problems, that is, reasoning in contexts of uncertainty. The purpose of this study is to describe an approach that allows assessing ill-defined problems and to present and discuss research findings related to this approach. The tool has been used up to now mainly in medicine, however it can be applied in all health professions. The approach is based on three principles: (a) examinees are faced with a challenging authenticsituation in which several options are relevant; (b) the response format is a Likert-type scale that reflects the way information is processed in problem-solving situations, according to the script theory; and (c) scoring is based on the aggregate scoring method to take into account the variability of reasoning processes among experts. Research findings suggest that the approach permits one to reliably discriminate examinees across their level of experience, and so in very different domains. It makes it possible to measure skills or domains that were up to now difficult to measure.

Development of clinical reasoning in an undergraduate medical program at a Brazilian university

CONTEXT AND OBJECTIVE The cognitive processes relating to the development of clinical reasoning are only partially understood, which explains the difficulties in teaching this skill in medical courses. This study aimed to understand how clinical reasoning develops among undergraduate medical students. DESIGN AND SETTING Quantitative and qualitative exploratory descriptive study conducted at the medical school of Universidade Federal de Goiás. METHODS The focus group technique was used among 40 students who participated in five focus groups, with eight students from each year, from the first to fifth year of the medical school program. The material was subjected to content analysis in categories, and was subsequently quantified and subjected to descriptive statistical analysis and chi-square test for inferential statistics. RESULTS The content of the students' statements was divided into two categories: clinical reasoning - in the preclinical phase, clinical reasoning was based on knowledge of basic medical science and in the clinical phase, there was a change to pattern recognition; knowledge of basic medical science - 80.6% of the students recognized its use, but they stated that they only used it in difficult cases. CONCLUSION In the preclinical phase, in a medical school with a traditional curriculum, clinical reasoning depends on the knowledge acquired from basic medical science, while in the clinical phase, it becomes based on pattern recognition.

Script concordance test in medical schools in Brazil: possibilities and limitations

CONTEXT AND OBJECTIVE

Routine use of the script concordance test (SCT) is not common in Brazilian universities. This study aimed to analyze application of the SCT in the medical school of a Brazilian university.

DESIGN AND SETTING

Quantitative, analytical and descriptive study in the medical school of a Brazilian university.

METHODS

A total of 159/550 students participated. The test comprised ten clinical cases within internal medicine, with five items per case, rated on a five-point Likert scale. The test was scored in accordance with a marking key that had been validated by a reference panel.

RESULTS

In the pre-clinical and clinical phases, the mean scores were 51.6% and 63.4% of the maximum possible scores, respectively. Comparison of the means of the responses among all the years showed that there were significant differences in 40% of the items. The panel marked all the possible answers in five items, while in one item, all the panelists marked a single answer. Cronbach's alpha was 0.64. The results indicated that the more senior students performed better. Construction of an SCT with discriminative questions was not easy. The low reliability index may have occurred due to: a) problems with the construction of the questions; b) limitations of the reference panel; and/or c) the scoring key.

CONCLUSION

This instrument is very difficult to construct, apply and correct. These difficulties may make application of an SCT as an assessment method unfeasible in units with limited resources.

The illusion of competency versus the desirability of expertise: seeking a common standard for support professions in sport

In this paper we examine and challenge the competency-based models which currently dominate accreditation and development systems in sport support disciplines, largely the sciences and coaching. Through consideration of exemplar shortcomings, the limitations of competency-based systems are presented as failing to cater for the complexity of decision making and the need for proactive experimentation essential to effective practice. To provide a better fit with the challenges of the various disciplines in their work with performers, an alternative approach is presented which focuses on the promotion, evaluation and elaboration of expertise. Such an approach resonates with important characteristics of professions, whilst also providing for the essential 'shades of grey' inherent in work with human participants. Key differences between the approaches are considered through exemplars of evaluation processes. The expertise-focused method, although inherently more complex, is seen as offering a less ambiguous and more positive route, both through more accurate representation of essential professional competence and through facilitation of future growth in proficiency and evolution of expertise in practice. Examples from the literature are also presented, offering further support for the practicalities of this approach.

Students' perceptions of the Script Concordance Test and its impact on their learning behavior: a mixed methods study

The Script Concordance Test (SCT) is increasingly used in postgraduate and undergraduate education as a method of summative clinical assessment. It has been shown to have high validity and reliability but there is little evidence of its use in veterinary education as assessment for learning. This study investigates some students' perceptions of the SCT and its effects on their approaches to learning. Final-year undergraduates of the School of Veterinary Medicine and Science (SVMS) at the University of Nottingham participated in a mixed-methods study after completing three formative SCT assessments. A qualitative, thematic analysis was produced from transcripts of three focus group discussions. The quantitative study was a survey based on the analyses of the qualitative study. Out of 50 students who registered for the study, 18 participated in the focus groups and 28 completed the survey. Clinical experience was regarded as the most useful source of information for answering the SCT. The students also indicated that recall of facts was perceived as useful for multiple-choice questions but least useful for the SCT. Themes identified in the qualitative study related to reliability, acceptability, educational impact, and validity of the SCT. The evidence from this study shows that the SCT has high face validity among veterinary students. They reported that it encouraged them to reflect upon their clinical experience, to participate in discussions of case material, and to adopt a deeper approach to clinical learning. These findings strongly suggest that the SCT is potentially a valuable method for assessing clinical reasoning and enhancing student learning.

The script concordance test for clinical reasoning: re-examining its utility and potential weakness

CONTEXT

The script concordance test (SCT) assesses clinical reasoning under conditions of uncertainty. Relatively little information exists on Z-score (standard deviation [SD]) cut-offs for distinguishing more experienced from less experienced trainees, and whether scores depend on factual knowledge. Additionally, a recent review highlighted the finding that the SCT is potentially weakened by the fact that the mere avoidance of extreme responses may greatly increase test scores.

OBJECTIVES

This study was conducted in order to elucidate the best cut-off Z-scores, to correlate SCT scores with scores on a separate medical knowledge examination (MKE), and to investigate potential solutions to the weakness of the SCT.

METHODS

An analysis of scores on pulmonary and critical care medicine tests undertaken during July and August 2013 was performed. Clinical reasoning was tested using 1-hour SCTs (Question Sets 1 or 2). Medical knowledge was tested using a 3-hour, computer-adapted, multiple-choice question examination.

RESULTS

The expert panel was composed of 16 attending physicians. The SCTs were completed by 16 fellows and 10 residents. Fourteen fellows completed the MKE. Test reliability was acceptable for both Question Sets 1 and 2 (Cronbach's alphas of 0.79 and 0.89, respectively). Z-scores of - 2.91 and - 1.76 best separated the scores of residents from those of fellows, and the scores of fellows from those of attending physicians, respectively. Scores on the SCT and MKE were poorly correlated. Simply avoiding extreme answers boosted the Z-scores of the lowest 10 scorers on both Question Sets 1 and 2 by ≥ 1 SD. Increasing the proportion of questions with extreme modal answers to 50%, and using hypothetical question sets created from Question Set 1 overcame this problem, but consensus scoring did not.

CONCLUSIONS

The SCT was able to differentiate between test subjects of varying levels of competence, and results were not associated with medical knowledge. However, the test was vulnerable to responses that intentionally avoided extreme values. Increasing the proportion of questions with extreme modal answers may attenuate the effect of candidates exploiting the test weakness related to extreme responses.

Assessing clinical reasoning: moving from in vitro to in vivo

The last century saw dramatic changes in clinical practice and medical education and the concomitant rise in high-stakes, psychometrically-based examinations of medical knowledge. Higher scores on these high-stakes “in-vitro” examinations are modestly associated with better performance in clinical practice and provide a meaningful degree of assurance to the public about physicians’ competency in medical knowledge. However, results on such examinations explain only a small fraction of the wide variation currently seen in clinical practice and diagnostic errors remain a serious and vexing problem for patients and the healthcare system despite decades of high-stakes examinations. In this commentary we explore some of the limitations of high-stakes examinations in assessing clinical reasoning and propose utilizing situated cognition theory to guide research and development of innovative modes of ”in-vivo” assessments that can be used in longitudinally and continuously in clinical practice.

Analyzing script concordance test scoring methods and items by difficulty and type

BACKGROUND

A battery of various psychometric assessments has been conducted on script concordance tests (SCTs) that are purported to measure data interpretation, an essential component of clinical reasoning. Although the breadth of published SCT research is broad, best practice controversies and evidentiary gaps remain.

PURPOSES

In this study, SCT data were used to test the psychometric properties of 6 scoring methods. In addition, this study explored whether SCT items clustered by difficulty and type were able to discriminate between medical training levels.

METHODS

SCT scores from a problem-solving SCT (SCT-PS; n = 522) and emergency medicine SCT (SCT-EM; n = 1,040) were collected at a large institution of medicine. Item analyses were performed to optimize each dataset. Items were categorized into difficulty levels and organized into types. Correlational analyses, one-way multivariate analysis of variance (MANOVA), repeated measures analysis of variance (ANOVA), and one-way ANOVA were conducted to explore study aims.

RESULTS

All 6 scoring methods differentiated between training levels. Longitudinal analysis of SCT-PS data reported that MS4s significantly (p < .001) outperformed their scores as MS2s in all difficulty categories. Cross-sectional analysis of SCT-EM data reported significant differences (p < .001) between experienced EM physicians, EM residents, and MS4s at each level of difficulty. Items categorized by type were also able to detect training level disparities.

CONCLUSIONS

Of the 6 scoring methods, 5-point scoring solutions generated more reliable measures of data interpretation than 3-point scoring methods. Data interpretation abilities were a function of experience at every level of item difficulty. Items categorized by type exhibited discriminatory power providing modest evidence toward the construct validity of SCTs.

Threats to validity in the use and interpretation of script concordance test scores

CONTEXT

Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high-stakes testing.

OBJECTIVES

This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats.

METHODS

We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice.

RESULTS

Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter-panel, inter-panellist and test-retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct-irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct-irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended.

CONCLUSIONS

Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

Standard setting of script concordance tests using an adapted Nedelsky approach

BACKGROUND

Standard setting in assessment seeks to apply meaning of achievement to an assessment score. Appropriate standard setting for script concordance tests (SCTs) remains a challenge, with existing methods representing norm-referenced approaches.

AIMS

To develop a criterion-referenced standard setting approach for sct using an adapted nedelsky approach, to pilot feasibility, and to compare failure rates with two other methods.

METHODS

Second- and third-year medical students were administered a 45-question SCT and results collated. Standard setting was applied using three approaches: (1) norm-referenced (student cohorts), (2) expert-referenced (student cohort compared to expert mean), and (3) adapted Nedelsky approach using answer key normalization. Feasibility and failure rates were measured.

RESULTS

All standard setting approaches were feasible, with 60 additional minutes required for the Nedelsky standard setting exercise. Failure rates between the three approaches were similar (Year 2: 8.0-9.8% and Year 3: 2.1-7.6%), with the adapted Nedelsky approach representing an intermediate option (Year 2: 8.0% and Year 3: 3.5%).

CONCLUSION

Standard setting SCT using the criterion-referenced method of an adapted Nedelsky approach was found to be both logically justifiable and logistically simple, and produced failure rates comparable to other currently utilized and less objective approaches.

Assessment of professionalism: a consolidation of current thinking

CONTEXT

Professionalism has become a hot topic in medical education. Professionalism needs to be assessed if it is to be viewed as both positive and relevant.

OBJECTIVES

The assessment of professionalism is an evolving field. This review aims to consolidate current thinking.

IMPLICATIONS

Assessment of professionalism has progressed from an initial focus on the development and attainment of professional identity, through identifying areas of deficiency, to the attainment of a set of identifiable positive attributes and behaviours. It is now beginning to recognise the challenge of assessing a multi-dimensional construct, looking beyond the measurement of behaviour to embrace a diversity of approaches.

CONCLUSIONS

Professionalism should be assessed longitudinally. It requires combinations of different approaches, assessing professionalism at individual, interpersonal and societal/institutional levels. Increasing the depth and the quality of reliability and validity of existing programmes in various contexts may be more appropriate than concentrating on developing new instruments. Increasing the number of tests and the number of relevant contexts will increase the reliability of the result. Similarly increasing the number of observers increases reliability. Feedback, encouraging reflection, can promote change in behaviour and identity formation.

Medical education research in the context of translational science

Health care struggles to transfer recent discoveries into high-quality medical care. Therefore, translational science seeks to improve the health of patients and communities by studying and promoting the translation of findings from bench research into clinical care. Similarly, medical education practice may be slow to adopt proven evidence of better learning and assessment. The Academic Emergency Medicine (AEM) consensus conference was designed to promote the dissemination of evidence-based education research and practice. We will pull from the work developed by the consensus conference as a means to create a roadmap for future medical education research using the framework of translational science.

Assessing diagnostic reasoning: a consensus statement summarizing theory, practice, and future needs

Assessment of an emergency physician (EP)'s diagnostic reasoning skills is essential for effective training and patient safety. This article summarizes the findings of the diagnostic reasoning assessment track of the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success." Existing theories of diagnostic reasoning, as they relate to emergency medicine (EM), are outlined. Existing strategies for the assessment of diagnostic reasoning are described. Based on a review of the literature, expert thematic analysis, and iterative consensus agreement during the conference, this article summarizes current assessment gaps and prioritizes future research questions concerning the assessment of diagnostic reasoning in EM.

Assessment of undergraduate clinical reasoning in geriatric medicine: application of a script concordance test

A challenging aspect of geriatric practice is that it often requires decision-making under conditions of uncertainty. The Script Concordance Test (SCT) is an assessment tool designed to measure clinical data interpretation, an important element of clinical reasoning under uncertainty. The purpose of this study was to develop and analyze the validity of results of an SCT administered to undergraduate students in geriatric medicine. An SCT consisting of 13 cases and 104 items covering a spectrum of common geriatric problems was designed and administered to 41 undergraduate medical students at a medical school in São Paulo, Brazil. A reference panel of 21 practicing geriatricians contributed to the test's score key. The responses were analyzed, and the psychometric properties of the tool were investigated. The test's internal consistency and discriminative capacity to distinguish students from experienced geriatricians supported construct validity. The Cronbach alpha for the test was 0.84, and mean scores for the experts were found to be significantly higher than those of the students (80.0 and 70.7, respectively; P < .001). This study demonstrated robust evidence of reliability and validity of an SCT developed for use in geriatric medicine for assessing clinical reasoning skills under conditions of uncertainty in undergraduate medical students. These findings will be of interest to those involved in assessing clinical competence in geriatrics and will have important potential application in medical school examinations.

Critical Reasoning Scores of Entering Bachelor’s and Master’s Students in an Occupational Therapy Program

We compared the critical reasoning (CR) of four classes of students entering a bachelor of occupational therapy program (n = 88) with the CR of five classes of students entering an entry-level master of occupational therapy program (n = 126) using the Watson–Glaser Critical Thinking Appraisal (WGCTA) and controlling for grade point average and reading comprehension as measured by the Nelson–Denny Reading Test. A multivariate analysis of covariance revealed a small but statistically significant difference between the groups’ CR scores. The univariate tests indicated that the groups differed with respect to their scores on one WGCTA subscale, Recognition of Assumptions; contrary to expectations, the bachelor’s students scored higher than the master’s students, although the effect size indicated small differences between the groups. Possible explanations for the findings and implications for occupational therapy education are discussed.

An innovative method to assess clinical reasoning skills: Clinical reasoning tests in the second national medical science Olympiad in Iran

BACKGROUND

Clinical reasoning plays a major role in the ability of doctors to make a diagnosis and reach treatment decisions. This paper describes the use of four clinical reasoning tests in the second National Medical Science Olympiad in Iran: key features (KF), script concordance (SCT), clinical reasoning problems (CRP) and comprehensive integrative puzzles (CIP). The purpose of the study was to design a multi instrument for multiple roles approach in clinical reasoning field based on the theoretical framework, KF was used to measure data gathering, CRP was used to measure hypothesis formation, SCT and CIP were used to measure hypothesis evaluation and investigating the combined use of these tests in the Olympiad. A bank of clinical reasoning test items was developed for emergency medicine by a scientific expert committee representing all the medical schools in the country. These items were pretested by a reference group and the results were analyzed to select items that could be omitted. Then 135 top-ranked medical students from 45 medical universities in Iran participated in the clinical domain of the Olympiad. The reliability of each test was calculated by Cronbach's alpha. Item difficulty and the correlation between each item and the total score were measured. The correlation between the students' final grade and each of the clinical reasoning tests was calculated, as was the correlation between final grades and another measure of knowledge, i.e., the students' grade point average.

RESULTS

The combined reliability for all four clinical reasoning tests was 0.91. Of the four clinical reasoning tests we compared, reliability was highest for CIP (0.91). The reliability was 0.83 for KF, 0.78 for SCT and 0.71 for CRP. Most of the tests had an acceptable item difficulty level between 0.2 and 0.8. The correlation between the score for each item and the total test score for each of the four tests was positive. The correlations between scores for each test and total score were highest for KF and CIP. The correlation between scores for each test and grade point average was low to intermediate for all four of the tests.

CONCLUSION

The combination of these four clinical reasoning tests is a reliable evaluation tool that can be implemented to assess clinical reasoning skills in talented undergraduate medical students, however these data may not generalizable to whole medical students population. The CIP and KF tests showed the greatest potential to measure clinical reasoning skills. Grade point averages did not necessarily predict performance in the clinical domain of the national competitive examination for medical school students.

Adjusting our lens: can developmental differences in diagnostic reasoning be harnessed to improve health professional and trainee assessment?

OBJECTIVES

Research in cognition has yielded considerable understanding of the diagnostic reasoning process and its evolution during clinical training. This study sought to determine whether or not this literature could be used to improve the assessment of trainees' diagnostic skill by manipulating testing conditions that encourage different modes of reasoning.

METHODS

The authors developed an online, vignette-based instrument with two sets of testing instructions. The "first impression" condition encouraged nonanalytic responses while the "directed search" condition prompted structured analytic responses. Subjects encountered six cases under the first impression condition and then six cases under the directed search condition. Each condition had three straightforward (simple) and three ambiguous (complex) cases. Subjects were stratified by clinical experience: novice (third- and fourth-year medical students), intermediate (postgraduate year [PGY] 1 and 2 residents), and experienced (PGY 3 residents and faculty). Two investigators scored the exams independently. Mean diagnostic accuracies were calculated for each group. Differences in diagnostic accuracy and reliability of the examination as a function of the predictor variables were assessed.

RESULTS

The examination was completed by 115 subjects. Diagnostic accuracy was significantly associated with the independent variables of case complexity, clinical experience, and testing condition. Overall, mean diagnostic accuracy and the extent to which the test consistently discriminated between subjects (i.e., yielded reliable scores) was higher when participants were given directed search instructions than when they were given first impression instructions. In addition, the pattern of reliability was found to depend on experience: simple cases offered the best reliability for discriminating between novices, complex cases offered the best reliability for discriminating between intermediate residents, and neither type of case discriminated well between experienced practitioners.

CONCLUSIONS

These results yield concrete guidance regarding test construction for the purpose of diagnostic skill assessment. The instruction strategy and complexity of cases selected should depend on the experience level and breadth of experience of the subjects one is attempting to assess.

Context and clinical reasoning: understanding the perspective of the expert's voice

OBJECTIVES

Prior work has found that a doctor's clinical reasoning performance varies on a case-by-case (situation) basis; this is often referred to as 'context specificity'. To explore the influence of context on diagnostic and therapeutic clinical reasoning, we constructed a series of videotapes to which doctors were asked to respond, modifying different contextual factors (patient, doctor, setting). We explored how these contextual factors, as displayed by videotape encounters, may have influenced the clinical reasoning of board-certified internists (experts). Our purpose was to clarify the influence of context on reasoning, to build upon education theory and to generate implications for education practice.

METHODS

Qualitative data about experts were gathered from two sources: think-aloud protocols reflecting concurrent thought processes that occurred while board-certified internists viewed videotape encounters, and free-text responses to queries that explicitly asked these experts to comment on the influence of selected contextual factors on their clinical reasoning processes. These data sources provided both actual performance data (think-aloud responses) and opinions on reflection (free-text answers) regarding the influence of context on reasoning. Results for each data source were analysed for emergent themes and then combined into a unified theoretical model.

RESULTS

Several themes emerged from our data and were broadly classified as components influencing the impact of contextual factors, mechanisms for addressing contextual factors, and consequences of contextual factors for patient care. Themes from both data sources had good overlap, indicating that experts are somewhat cognisant of the potential influences of context on their reasoning processes; notable exceptions concerned the themes of missed key findings, balancing of goals and the influence of encounter setting, which emerged in the think-aloud but not the free-text analysis.

CONCLUSIONS

Our unified model is consistent with the tenets of cognitive load, situated cognition and ecological psychology theories. A number of potentially modifiable influences on clinical reasoning were identified. Implications for doctor training and practice are discussed.

Script concordance testing: a review of published validity evidence

CONTEXT

Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established.

OBJECTIVES

This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method.

METHODS

We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences.

RESULTS

Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences.

CONCLUSIONS

Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

Integrated Cases Section: a course designed to promote clinical reasoning in year 2 medical students

BACKGROUND

Medical students often have difficulty applying basic science knowledge within clinical contexts as they transition into clerkships.

DESCRIPTION

To enhance clinical reasoning skills in 2nd-year medical students, we developed a 7-week transition course called the Integrated Cases Section. Curricular instruction incorporated analytic and nonanalytic clinical reasoning models. Practice with variable case scenarios enhanced students' application of basic science knowledge to clinical problem solving.

EVALUATION

We evaluated curricular design and objectives by measuring student perceptions during the course and following completion of 2 clerkship rotations. To obtain measurement of students' clinical reasoning ability we administered a script concordance test immediately before and after the course.

CONCLUSIONS

Students reported increased confidence in their diagnostic reasoning ability during the course and after completion of 2 clerkships. Students' clinical reasoning showed a significant gain after the Integrated Cases Section on a script concordance test. Student support has solidified Integrated Cases Section in the curriculum.

Assessment of intraoperative judgment during gynecologic surgery using the Script Concordance Test

OBJECTIVE

We sought to develop a valid, reliable assessment of intraoperative judgment by residents during gynecologic surgery based on Script Concordance Theory.

STUDY DESIGN

This was a multicenter prospective study involving 5 obstetrics and gynecology residency programs. Surgeons from each site generated case scenarios based on common gynecologic procedures. Construct validity was evaluated by correlating scores to training level, in-service examinations, and surgical skill and experience using a Global Rating Scale of Operative Performance and case volumes.

RESULTS

A final test that included 42 case scenarios was administered to 75 residents. Internal consistency (Cronbach alpha = 0.73) and test-retest reliability (Lin correlation coefficient = 0.76) were good. There were significant differences between test scores and training levels (P = .002) and test scores correlated with in-service examination scores (r = 0.38; P = .001). There was no association between test scores and total number of cases or technical skills.

CONCLUSION

The Script Concordance Test appears to be a reliable, valid assessment tool for intraoperative decision-making during gynecologic surgery.

Assessment in the context of uncertainty using the script concordance test: more meaning for scores

BACKGROUND

The Script Concordance Test (SCT) uses authentic, ill-defined clinical cases to compare medical learners' judgment skills with those of experienced physicians. SCT scores are meant to measure the degree of concordance between the performance of examinees and that of the reference panel. Raw test scores have meaning only if statistics (mean and standard deviation) describing the panel's performance are concurrently provided.

PURPOSE

The purpose of this study is to suggest a method for reporting scores that standardizes panel mean and standard deviation, allowing examinees to immediately gauge their performance relative to panel members.

METHODS

Based on a statistical method of standardization, a new method for computing SCT scores is described. According to this method, test raw scores are converted into a scale in which the panel mean is set as the value of reference, and the standard deviation of the panel serves as a yardstick by which examinee performance is measured.

RESULTS

The effect of this transformation on four data sets obtained from SCTs in radio-oncology, surgery, neurology, and nursing is discussed.

CONCLUSION

This transformation method proposes a common metric basis for reporting SCT scores and provides examinees with clear, interpretable insights into their performance relative to that of physicians of the field. We recommend reporting SCT scores with the mean and standard deviation of panel scores set at standard scores of 80 and 5, respectively. Beyond SCT, our transformation method may be generalizable to the scoring of other test formats in which the performance of examinees and those of a panel of reference undertaking the same cognitive tasks are compared.

Teaching and learning professional behavior in practice

This paper is the fourth article in a series on Professionalism and provides an overview of current methods used for teaching and learning about professionalism. The questions "whether" and "how" professionalism can be placed in the formal medical school curricula are addressed, and the informal learning related to professionalism reviewed.

General considerations regarding assessment of professional behaviour

Core medical knowledge has been assessed for over fifty years and technical and communication skills for at least twenty. The assessment of professionalism however has failed to achieve recognition within this time frame. The interest in the assessment of professionalism and professional behaviour thus is a fairly recent development. This article will firstly clarify how professional behaviour assessment relates to other assessment methods using the framework proposed by Miller6. Thereafter a brief overview will be provided of the current "tool box" of methods available to assess professionalism. Data on the validity, reliability, feasibility, acceptability and educational utility of these "tools" as derived from published evidence will be reviewed. Subsequently a general overview of the way forward in the assessment of professionalism and professional behaviour will be given.

Is a web-based concordance test feasible to assess therapeutic decision-making skills in a French context?

BACKGROUND

At the end of undergraduate medical curriculum, a written simulation-based examination is used in France to assess therapeutic decision-making skills and to rank students for the purpose of matching their training specialties. However, this examination based on a single assessment method remains a subject of debate.

AIM

To study the feasibility of a web-based Concordance test for therapeutic decision-making assessment.

METHODS

A 12 clinical-case Concordance test was developed based on objectives for the undergraduate training program. The test was administered on line to candidates with different levels of clinical experience. Fifteen therapeutic teachers constituted the reference panel. Data analysis included analysis of variance, post-hoc test, and Cronbach's alpha.

RESULTS

One hundred and seventy participants (113 students, 34 residents, 23 physicians) fully completed the free-access test on line with no technical problems. Differences between the mean scores for groups were significant (p < 0.001). Significant differences occurred between fourth year students and residents (p < 0.001), fourth year students and physicians (p = 0.001). No difference was found between residents and physicians. Reliability coefficient was 0.67.

CONCLUSION

A web-based Concordance test in the field of therapeutic decision-making was considered feasible in a French learning environment. Further research is warranted to determine its usefulness as a part of the National Examination.

Assessing clinical reasoning in pediatric emergency medicine: validity evidence for a Script Concordance Test

STUDY OBJECTIVE

Clinical reasoning is a crucial skill for all residents to acquire during their training. During most patient encounters in pediatric emergency medicine, physicians and trainees are challenged by diagnostic, investigative, and treatment uncertainties. The Script Concordance Test may provide a means to assess reasoning skills in the context of uncertainty in the practice of pediatric emergency medicine. We gathered validity evidence for the use of a pediatric emergency medicine Script Concordance Test to evaluate residents' reasoning skills.

METHODS

A 1-hour test containing 60 questions nested in 38 cases was administered to 53 residents at the end of their pediatric emergency medicine rotation at 1 academic institution. Twelve experienced pediatricians were part of a reference panel to establish the basis for the scoring process.

RESULTS

An optimized version of the test, based on positive item discrimination data, contained 30 cases and 50 questions. Scores ranged from 48% to 82%, with a mean score of 69.9 (SD=11.5). The reliability of the optimized test (Cronbach's alpha) was 0.77. Performance on the test increased as the level of experience of the residents increased. The residents considered the Script Concordance Test true to real-life clinical problems and had enough time to complete the test.

CONCLUSION

This pediatric emergency medicine Script Concordance Test was reliable and useful to assess the progression of clinical reasoning during residency training.

Does training family physicians in shared decision making promote optimal use of antibiotics for acute respiratory infections? Study protocol of a pilot clustered randomised controlled trial

BACKGROUND

In North America, although it varies according to the specific type of acute respiratory infections (ARI), use of antibiotics is estimated to be well above the expected prevalence of bacterial infections. The objective of this pilot clustered randomized controlled trial (RCT) is to assess the feasibility of a larger clustered RCT aiming at evaluating the impact of DECISION+, a continuing professional development (CPD) program in shared decision making, on the optimal use of antibiotics in the context of ARI.

METHODS/DESIGN

This pilot study is a cluster RCT conducted with family physicians from Family Medicine Groups (FMG) in the Quebec City area, Canada. Participating FMG are randomised to an immediate DECISION+ group, a CPD program in shared decision making, (experimental group), or a delayed DECISION+ group (control group). Data collection involves recruiting five patients consulting for ARI per physician from both study groups before (Phase 1) and after (Phase 2) exposure of the experimental group to the DECISION+ program, and after exposure of the control group to the DECISION+ program (Phase 3). The primary outcome measures to assess the feasibility of a larger RCT include: 1) proportion of contacted FMG that agree to participate; 2) proportion of recruited physicians who participate in the DECISION+ program; 3) level of satisfaction of physicians regarding DECISION+; and 4) proportion of missing data in each data collection phase. Levels of agreement of the patient-physician dyad on the Decisional Conflict Scale and physicians' prescription profile for ARI are performed as secondary outcome measures.

DISCUSSION

This study protocol is informative for researchers and clinicians interested in designing and/or conducting clustered RCT with FMG regarding training of physicians in shared decision making.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT00354315.

Preliminary evaluation of the Web Initiative for Surgical Education (WISE-MD)

BACKGROUND

Major changes in health care delivery and financing have negatively impacted students' experience during the surgery clerkship, particularly their exposure to physicians' decision-making processes and to the continuity of patient care. In response to these dilemmas in surgical education, we have developed the Web Initiative for Surgical Education (WISE-MD), a comprehensive surgery clerkship curriculum delivered through multimedia teaching modules and designed to enhance exposure to surgical disease and clinical reasoning.

METHODS

As part of the process of creating WISE-MD, we conducted preliminary studies to assess the impact of this computer-assisted approach on students' knowledge, clinical reasoning, and satisfaction.

RESULTS

Compared to students who did not view the modules, early data show a trend toward improved knowledge and an improvement in clinical reasoning for students who used the WISE-MD modules. This effect was specific to the clinical content area addressed in the module seen by the students. Most students felt the module was superior to traditional teaching methods and enhanced their understanding of surgical technique and anatomy.

CONCLUSIONS

WISE-MD, a theory-driven example of a concerted technology-based approach to surgical education, has the potential to address the myriad problems of today's clinical learning environment.