Development of a reliable multimedia, computer-based measure of clinical skills in bedside neurology

Effectiveness of a modified Continuum curriculum for medical students: a randomized trial

BACKGROUND

Continuum: Lifelong Learning in Neurology® is a well-regarded and widely used continuing medical education tool published by the American Academy of Neurology. The objective of this study was to test the effectiveness of a modified version of the Dementia module of the Continuum curriculum, adapted for medical students rotating on their neurology clerkship, in increasing medical knowledge of dementia.

METHODS

A multisite longitudinal randomized controlled design was used. Medical students rotating on their Neurology clerkship were recruited from 2 US medical schools. Participants completed 10 multiple-choice questions, 1 fill-in-the-blank, and 1 patient case simulation question to assess medical knowledge of the most prevalent dementias pre- and post-curriculum implementation. All students received their standard dementia curriculum (45-minute live didactic presentation on dementia along with a copy of the slides in handout form). Students were randomized to either the intervention (standard + Continuum curriculum) or control (standard curriculum alone) group. Data collection and outcomes assessment was optimized via an interactive audience response system (pretest) and Web-based survey/database tool (post-test and student satisfaction surveys).

RESULTS

From pre- to post-clinical clerkship, medical students completing the Continuum dementia curriculum in addition to standard clerkship curriculum demonstrated significantly greater increases in medical knowledge of dementia, relative to students completing only the standard curriculum. Subscores were significantly higher among Continuum-trained students on questions regarding Alzheimer disease (AD), frontotemporal lobar dementia, Lewy body dementia, AD treatment fill-in-the-blank, and AD patient case simulation.

CONCLUSIONS

The Continuum: Dementia for Medical Students curriculum provided an inexpensive and readily implementable means for improving medical knowledge of dementia. Improved performance on an AD patient case simulation may be considered a surrogate marker for optimized patient care.

The Ventriloscope® as an innovative tool for assessing clinical examination skills: appraisal of a novel method of simulating auscultatory findings

BACKGROUND

Simulation is increasingly used as a teaching tool and in assessment. The Ventriloscope® (VS) is a new auscultation simulator. This modified stethoscope allows pre-recorded sounds (activated wirelessly) to be integrated with a simulated patient (SP, professional actor).

AIMS

This study explores the instrument's potential for overcoming limitations of current objective structured clinical examination (OSCE) assessment by increasing validity while retaining reliability.

METHODS

After training SPs to synchronise the device with their breathing (recreating abnormal signs), we evaluated the VS during a third year undergraduate medical student OSCE. Students (n =  385), examiners (n = 19) and SPs (n = 10) completed post-exam questionnaires which were analysed using a coding framework. OSCE performance data were analysed using Stata 10.

RESULTS

When 'compared to their usual stethoscope' 40% of students found no difference in using the VS; 69% found it easier to identify sounds; 68% found examination with the VS very or fairly realistic when 'compared to examining a real patient'. Examination scores were comparable with other OSCE stations.

CONCLUSIONS

The VS reliably provided consistent 'abnormal' auscultatory signs within an OSCE framework. Using a VS may increase OSCE validity, allowing examiners to assess students' application of knowledge in a realistically simulated setting. The VS can help bridge the gap between simulation and real patients.

Does physical examination competence correlate with bedside diagnostic acumen? An observational study

AIM

To examine the relationship between a physician's ability to examine a standardized patient (SP) and their ability to correctly identify related clinical findings created with simulation technology.

METHOD

The authors conducted an observational study of 347 candidates during a Canadian national specialty examination at the end of post-graduate internal medicine training. Stations were created that combined physical examination of an SP with evaluation of a related audio-video simulation of a patient abnormality, in the domains of cardiology and neurology. Examiners evaluated a candidate's competence at performing a physical examination of an SP and their accuracy in diagnosing a related audio-video simulation.

RESULTS

For the cardiology stations, the correlation between the physical examination scores and recognition of simulation abnormalities was 0.31 (p < 0.01). For the neurology stations, the correlation was 0.27 (p < 0.01). Addition of the simulations identified 18% of 197 passing candidates on the cardiology stations and 17% of 240 passing candidates on the neurology stations who were competent in their physical examination technique but did not achieve the passing score for diagnostic skills.

CONCLUSIONS

Assessments incorporating SPs without physical findings may need to include other methodologies to assess bedside diagnostic acumen.

Evaluation of the Clinical Assessment project: a computer-based multimedia tool to assess problem-solving ability in medical students

OBJECTIVE

The purpose of this work was to describe Clinical Assessment, a computer-based multimedia patient simulation used to assess the problem-solving abilities of medical students and to evaluate its capacity to guide the assignment of course grade.

METHODS

This was a multisite reviewer-blinded comparison of course grades, National Board of Medical Examiners pediatric examination score, and Clinical Assessment scores at 3 pediatric clerkship sites of the Harvard Medical School. Participants included 470 students completing their pediatric clerkships. Each student's performance on < or = 4 Clinical Assessment patient case simulations was compared with National Board of Medical Examiners pediatric examination scores and course grades assigned by clerkship directors based on overall ward performance.

RESULTS

Data from both the National Board of Medical Examiners pediatric "shelf" examination and the course grade were available for 411 students who completed > or = 1 Clinical Assessment case. There was a strong correlation between Clinical Assessment score and course grade when comparing students receiving honors versus satisfactory category course grades. Students who ordered more expensive or greater numbers of laboratory tests did not achieve greater diagnostic accuracy on Clinical Assessment. Clinical Assessment had a high positive predictive value for course grade: 95% of students scoring > or = 90% on Clinical Assessment achieved an honors category course grade.

CONCLUSIONS

Because nearly all of the students who scored very well on Clinical Assessment received honors category course grades, future high scorers on this examination merit consideration for assigning a high course grade. A computer-based multimedia patient simulation assessment tool provides objective information that can complement a student's National Board of Medical Examiners score and course grade and may assist in evaluating clinical problem-solving ability.

A multicenter study to provide evidence of construct validity in a computer-based outcome measure of neurology clinical skills

BACKGROUND

Using computer-based simulation to assess clinical skill-a key competence for medical trainees-enables standardization and exposure to a broad sample of physical findings. The purpose of this study is to provide evidence of construct validity for a computer-based outcome measure of neurology clinical skills.

METHOD

A total of 128 medical students and neurology residents at four institutions volunteered to take a 34-question computer-based test designed to measure neurology clinical skills. Subjects were classified into three groups based on level of training: novice, intermediate, and experienced.

RESULTS

Overall performance increased with level of training. Question difficulty discriminated between groups as predicted. Twenty-six of 34 individual items discriminated between novices and more advanced learners. This test separated learners at different levels of training with a consistency of .92.

CONCLUSION

This study provided evidence of construct validity for a computer-based outcome measure of neurology clinical skills.

Incorporating simulation technology in a canadian internal medicine specialty examination: a descriptive report

High-stakes assessment of clinical performance through the use of standardized patients (SPs) is limited by the SP's lack of real physical abnormalities. The authors report on the development and implementation of physical examination stations that combine simulation technology in the form of digitized cardiac auscultation videos with an SP assessment for the 2003 Royal College of Physicians and Surgeons of Canada's Comprehensive Objective Examination in Internal Medicine. The authors assessed candidates on both the traditional stations and the stations that combined the traditional SP examination with the digitized cardiac auscultation video. For the combined stations, candidates first completed a physical examination of the SP, watched and listened to a computer simulation, and then described their auscultatory findings. The candidates' mean scores for both types of stations were similar, as were the mean discrimination indices for both types of stations, suggesting that the combined stations were of a testing standard similar to the traditional stations. Combining an SP with simulation technology may be one approach to the assessment of clinical competence in high-stakes testing situations.

Trends in medical education research

The medical education community is reflecting increasingly on the role and nature of research in the field. Useful sources of data to include in these reflections are a description of the topics in which we are investing our energies, an analysis of the extent to which there is a sense of progress on these topics, and an examination of the mechanisms by which any progress has been achieved. This article presents the results of a thematic review of the medical education research literature in four key journals since the turn of the 21st century. It describes four examples of areas in which the community appears to be investing its energies: curriculum and teaching issues, skills and attitudes relevant to the structure of the profession, individual characteristics of medical students, and the evaluation of students and residents. A discussion of the recent publications in these domains highlights a distinction between thematic categories of research, in which many members of the community are working on the same topic, and programmatic lines of research, in which members of the community are working together toward the shared goal of consensual understanding. The author suggests that community-level, programmatic lines of research are necessary to build knowledge and understanding of a domain and that, in the absence of such communal effort, the value of research is limited to the uncoordinated accrual of information.