Improving diagnostic capabilities of medical students via application of cognitive sciences-derived learning principles

Faculty Development Directed at Curricular Reforms Designed to Improve Patient Outcomes

Three initiatives involving quality of patient outcomes that evolved in the late 1990s must be considered in the design of 21st century undergraduate medical curricula. They involve (1) the question of how to best teach and assess medical competencies, (2) growing concerns regarding the frequency and severity of error in medical care, and (3) the role physicians might play in weaving together the overlapping elements of population-, community-, and systems-based practice into a codified approach to medical care. With these initiatives in mind, the University of North Texas Health Science Center Texas College of Osteopathic Medicine has formed an Academy of Medical Educators whose goal is to develop faculty programs intended to expedite curricular modifications and reforms.

Comparison of the Effectiveness of Interactive Didactic Lecture Versus Online Simulation-Based CME Programs Directed at Improving the Diagnostic Capabilities of Primary Care Practitioners

INTRODUCTION

Diagnostic errors in primary care contribute to increased morbidity and mortality, and billions in costs each year. Improvements in the way practicing physicians are taught so as to optimally perform differential diagnosis can increase patient safety and lower the costs of care. This study represents a comparison of the effectiveness of two approaches to CME training directed at improving the primary care practitioner's diagnostic capabilities against seven common and important causes of joint pain.

METHODS

Using a convenience sampling methodology, one group of primary care practitioners was trained by a traditional live, expert-led, multimedia-based training activity supplemented with interactive practice opportunities and feedback (control group). The second group was trained online with a multimedia-based training activity supplemented with interactive practice opportunities and feedback delivered by an artificial intelligence-driven simulation/tutor (treatment group).

RESULTS

Before their respective instructional intervention, there were no significant differences in the diagnostic performance of the two groups against a battery of case vignettes presenting with joint pain. Using the same battery of case vignettes to assess postintervention diagnostic performance, there was a slight but not statistically significant improvement in the control group's diagnostic accuracy (P = .13). The treatment group, however, demonstrated a significant improvement in accuracy (P < .02; Cohen d, effect size = 0.79).

DISCUSSION

These data indicate that within the context of a CME activity, a significant improvement in diagnostic accuracy can be achieved by the use of a web-delivered, multimedia-based instructional activity supplemented by practice opportunities and feedback delivered by an artificial intelligence-driven simulation/tutor.

Evidence of the preferential use of disease prototypes over case exemplars among early year one medical students prior to and following diagnostic training

Two core dual processing theory (DPT) System I constructs (Exemplars and Prototypes) were used to: 1) formulate a training exercise designed to improve diagnostic performance in year one medical students, and 2) explore whether any observed performance improvements were associated with preferential use of exemplars or prototypes.

With IRB approval, 117 year one medical students participated in an acute chest pain diagnostic training exercise. A pre- and post-training test containing the same 27 case vignettes was used to determine if the subjects’ diagnostic performance improved via training in both exemplars and prototypes. Exemplar and Prototype theory was also used to generate a unique typicality estimate for each case vignette. Because these estimates produce different performance predictions, differences in the subjects’ observed performance would make it possible to infer whether subjects were preferentially using Exemplars or Prototypes.

Pre- vs. post-training comparison revealed a significant performance improvement; t=14.04, p<0.001, Cohen’s d=1.32. Pre-training, paired t-testing demonstrated that performance against the most typical vignettes>mid typical vignettes: t=4.94, p<0.001; and mid typical>least typical: t=5.16, p<0.001. Post-training, paired t-testing again demonstrated that performance against the most typical vignettes>mid typical: t=2.94, p<0.01; and mid typical>least typical: t=6.64, p<0.001. These findings are more consistent with the performance predictions generated via Prototype theory than Exemplar theory.

DPT is useful in designing and evaluating the utility of new approaches to diagnostic training, and, investigating the cognitive factors driving diagnostic capabilities among early medical students.

How to improve the teaching of clinical reasoning: a narrative review and a proposal

CONTEXT

The development of clinical reasoning (CR) in students has traditionally been left to clinical rotations, which, however, often offer limited practice and suboptimal supervision. Medical schools begin to address these limitations by organising pre-clinical CR courses. The purpose of this paper is to review the variety of approaches employed in the teaching of CR and to present a proposal to improve these practices.

METHODS

We conducted a narrative review of the literature on teaching CR. To that end, we searched PubMed and Web of Science for papers published until June 2014. Additional publications were identified in the references cited in the initial papers. We used theoretical considerations to characterise approaches and noted empirical findings, when available.

RESULTS

Of the 48 reviewed papers, only 24 reported empirical findings. The approaches to teaching CR were shown to vary on two dimensions. The first pertains to the way the case information is presented. The case is either unfolded to students gradually - the 'serial-cue' approach - or is presented in a 'whole-case' format. The second dimension concerns the purpose of the exercise: is its aim to help students acquire or apply knowledge, or is its purpose to teach students a way of thinking? The most prevalent approach is the serial-cue approach, perhaps because it tries to directly simulate the diagnostic activities of doctors. Evidence supporting its effectiveness is, however, lacking. There is some empirical evidence that whole-case, knowledge-oriented approaches contribute to the improvement of students' CR. However, thinking process-oriented approaches were shown to be largely ineffective.

CONCLUSIONS

Based on research on how expertise develops in medicine, we argue that students in different phases of their training may benefit from different approaches to the teaching of CR.

The balance beam metaphor: a perspective on clinical diagnosis

Understanding the impact of clinical findings in discriminating between possible causes of a patient's presentation is essential in clinical judgment. A balance beam is a natural physical analogue that can accurately represent the combination of several pieces of evidence with varying ability to discriminate between disease hypotheses. Calculation of Bayes' theorem using log(posterior odds) as a function of log(prior odds) and the logarithms of the evidence's likelihood ratios maps onto the physical forces affecting objects placed on a balance beam. We describe the rules governing the functioning of tokens representing clinical findings in the comparison of 2 competing diseases. The likelihood ratios corresponding to positive (LR+) or negative (LR-) observations for each symptom determine the lateral position at which the symptom's token is placed on the beam, using a weight if the finding is present and a helium balloon if it is absent. We discuss how a balance beam could represent concepts of dynamic specificity (due to changes in competitor diseases' probabilities) and dynamic sensitivity (due to class-conditional independence). Utility-based thresholds for acting on a diagnosis could be represented by moving the balance beam's fulcrum. It is suggested that a balance beam can be a useful aid for students learning clinical diagnosis, allowing them to build on existing intuitive understanding to develop an appreciation of how evidence combines to influence degree of belief. The balance beam could also facilitate exploration of the potential impact of available questions or investigations.

A balance beam aid for instruction in clinical diagnostic reasoning

We describe a balance beam aid for instruction in diagnosis (BBAID) and demonstrate its potential use in supplementing the training of medical students to diagnose acute chest pain. We suggest the BBAID helps students understand the process of diagnosis because the impact of tokens (weights and helium balloons) attached to a beam at different distances from the fulcrum is analogous to the impact of evidence to the relative support for 2 diseases. The BBAID presents a list of potential findings and allows students to specify whether each is present, absent, or unknown. It displays the likelihood ratios corresponding to a positive (LR+) or negative (LR-) observation for each symptom, for any pair of diseases. For each specified finding, a token is placed on the beam at a location whose distance from the fulcrum is proportional to the finding's log(LR): a downward force (a weight) if the finding is present and a lifting force (a balloon) if it is absent. Combining the physical torques of multiple tokens is mathematically identical to applying Bayes' theorem to multiple independent findings, so the balance beam is a high-fidelity metaphor. Seven first-year medical students and 3 faculty members consulted the BBAID while diagnosing brief patient case vignettes. Student comments indicated the program is usable, helpful for understanding pertinent positive and negative findings' usefulness in particular situations, and welcome as a reference or self-test. All students attended the effect of the tokens on the beam, although some stated they did not use the numerical statistics. Faculty noted the BBAID might be particularly helpful in reminding students of diseases that should not be missed and identifying pertinent findings to ask for.

Learning sciences principles that can inform the construction of new approaches to diagnostic training

The author suggests that the ill-defined nature of human diseases is a little appreciated, nonetheless important contributor to persistent and high levels of diagnostic error. Furthermore, medical education’s continued use of traditional, non-evidence based approaches to diagnostic training represents a systematic flaw likely perpetuating sub-optimal diagnostic performance in patients suffering from ill-defined diseases. This manuscript briefly describes how Learning Sciences findings elucidating how humans reason in the face of the uncertainty and complexity posed by ill-defined diseases might serve as guiding principles in the formulation of first steps towards a codified, 21st century approach to training and assessing the diagnostic capabilities of future health care providers.

How can students' diagnostic competence benefit most from practice with clinical cases? The effects of structured reflection on future diagnosis of the same and novel diseases

PURPOSE

To develop diagnostic competence, students should practice with many examples of clinical problems to build rich mental representations of diseases. How to enhance learning from practice remains unknown. This study investigated the effects of reflection on cases compared with generating a single or differential diagnosis.

METHOD

In 2012, during the learning phase, 110 fourth-year medical students diagnosed four cases of two criterion diseases under three different experimental conditions: structured reflection, single-diagnosis, or differential-diagnosis. One week later, they diagnosed two novel exemplars of each criterion disease and four cases of new diseases that were not among the cases of the learning phase but were plausible alternative diagnoses.

RESULTS

Diagnostic performance did not differ among the groups in the learning phase. One week later, the reflection group obtained higher mean diagnostic accuracy scores (range: 0-1) than the other groups when diagnosing new exemplars of criterion diseases (reflection: 0.67; single-diagnosis: 0.36, P < .001; differential-diagnosis: 0.51, P = .014) and cases of new diseases (reflection: 0.44; single-diagnosis: 0.32, P = .010; differential-diagnosis: 0.33, P = .015). No difference was found between the single-diagnosis and the differential-diagnosis conditions.

CONCLUSIONS

Structured reflection while practicing with cases enhanced learning of diagnosis both of the diseases practiced and of their alternative diagnoses, suggesting that reflection not only enriched mental representations of diseases practiced relative to more conventional approaches to clinical learning but also influenced the representations of adjacent but different diseases. Structured reflection seems a useful addition to the existing clinical teaching methods.

Analysing clinical reasoning characteristics using a combined methods approach

BACKGROUND

Despite a major research focus on clinical reasoning over the last several decades, a method of evaluating the clinical reasoning process that is both objective and comprehensive is yet to be developed.The aim of this study was to test whether a dual approach, using two measures of clinical reasoning, the Clinical Reasoning Problem (CRP) and the Script Concordance Test (SCT), provides a valid, reliable and targeted analysis of clinical reasoning characteristics to facilitate the development of diagnostic thinking in medical students.

METHODS

Three groups of participants, general practitioners, and third and fourth (final) year medical students completed 20 on-line clinical scenarios -10 in CRP and 10 in SCT format. Scores for each format were analysed for reliability, correlation between the two formats and differences between subject-groups.

RESULTS

Cronbach's alpha coefficient ranged from 0.36 for SCT 1 to 0.61 for CRP 2, Statistically significant correlations were found between the mean f-score of the CRP 2 and total SCT 2 score (0.69); and between the mean f-score for all CRPs and all mean SCT scores (0.57 and 0.47 respectively). The pass/fail rates of the SCT and CRP f-score are in keeping with the findings from the correlation analysis (i.e. 31% of students (11/35) passed both, 26% failed both, and 43% (15/35) of students passed one but not the other test), and suggest that the two formats measure overlapping but not identical characteristics. One-way ANOVA showed consistent differences in scores between levels of expertise with these differences being significant or approaching significance for the CRPs.

CONCLUSION

SCTs and CRPs are overlapping and complementary measures of clinical reasoning. Whilst SCTs are more efficient to administer, the use of both measures provides a more comprehensive appraisal of clinical skills than either single measure alone, and as such could potentially facilitate the customised teaching of clinical reasoning for individuals. The modest reliability of SCTs and CRPs in this study suggests the need for an increased number of items for testing. Further work is needed to determine the suitability of a combined approach for assessment purposes.

Time and learning efficiency in Internet-based learning: a systematic review and meta-analysis

Authors have claimed that Internet-based instruction promotes greater learning efficiency than non-computer methods.

OBJECTIVES

determine, through a systematic synthesis of evidence in health professions education, how Internet-based instruction compares with non-computer instruction in time spent learning, and what features of Internet-based instruction are associated with improved learning efficiency.

DATA SOURCES

we searched databases including MEDLINE, CINAHL, EMBASE, and ERIC from 1990 through November 2008. STUDY SELECTION AND DATA ABSTRACTION we included all studies quantifying learning time for Internet-based instruction for health professionals, compared with other instruction. Reviewers worked independently, in duplicate, to abstract information on interventions, outcomes, and study design.

RESULTS

we identified 20 eligible studies. Random effects meta-analysis of 8 studies comparing Internet-based with non-Internet instruction (positive numbers indicating Internet longer) revealed pooled effect size (ES) for time -0.10 (p = 0.63). Among comparisons of two Internet-based interventions, providing feedback adds time (ES 0.67, p =0.003, two studies), and greater interactivity generally takes longer (ES 0.25, p = 0.089, five studies). One study demonstrated that adapting to learner prior knowledge saves time without significantly affecting knowledge scores. Other studies revealed that audio narration, video clips, interactive models, and animations increase learning time but also facilitate higher knowledge and/or satisfaction. Across all studies, time correlated positively with knowledge outcomes (r = 0.53, p = 0.021).

CONCLUSIONS

on average, Internet-based instruction and non-computer instruction require similar time. Instructional strategies to enhance feedback and interactivity typically prolong learning time, but in many cases also enhance learning outcomes. Isolated examples suggest potential for improving efficiency in Internet-based instruction.