Reliability of performance-based clinical skill assessment of emergency medicine residents

Emergency Radiology "Boot Camp": Educating Emergency Medicine Residents Using E-learning Radiology Modules

OBJECTIVES

There is an overall paucity of literature on the radiologic education of emergency medicine (EM) clinicians. Given the fact that many EM clinicians preliminarily review images for their patients, we hypothesized that a brief imaging curriculum could be efficacious in teaching basic and relevant radiologic interpretation.

METHODS

We designed a 4-hour "radiology boot camp" for a group of 20 EM residents (from all years of training) covering several subject specific e-learning modules. They completed precourse and postcourse quizzes to evaluate the efficacy of these modules. These modules included interactive PowerPoint-based tutorials, games, and imaging decision support simulators. Matched results from the pre- and posttests were analyzed using paired t test. An additional questionnaire was administered to the EM residents to evaluate their perception of the educational experience.

RESULTS

The precourse and postcourse quizzes demonstrated a statistically significant level of improved knowledge due to the educational modules (p < 0.0001). In addition, all of the participants believed the modules were a useful learning experience (100%) and a vast majority described them as a valuable resource for future reference (95%).

CONCLUSION

We demonstrate a model for providing an easy and effective means of educating EM residents on basic imaging interpretation and utilization, using e-learning modules.

Research Priorities in the Utilization and Interpretation of Diagnostic Imaging: Education, Assessment, and Competency

The appropriate selection and accurate interpretation of diagnostic imaging is a crucial skill for emergency practitioners. To date, the majority of the published literature and research on competency assessment comes from the subspecialty of point-of-care ultrasound. A group of radiologists, physicists, and emergency physicians convened at the 2015 Academic Emergency Medicine consensus conference to discuss and prioritize a research agenda related to education, assessment, and competency in ordering and interpreting diagnostic imaging. A set of questions for the continued development of an educational curriculum on diagnostic imaging for trainees and competency assessment using specific assessment methods based on current best practices was delineated. The research priorities were developed through an iterative consensus-driven process using a modified nominal group technique that culminated in an in-person breakout session. The four recommendations are: 1) develop a diagnostic imaging curriculum for emergency medicine (EM) residency training; 2) develop, study, and validate tools to assess competency in diagnostic imaging interpretation; 3) evaluate the role of simulation in education, assessment, and competency measures for diagnostic imaging; 4) study is needed regarding the American College of Radiology Appropriateness Criteria, an evidence-based peer-reviewed resource in determining the use of diagnostic imaging, to maximize its value in EM. In this article, the authors review the supporting reliability and validity evidence and make specific recommendations for future research on the education, competency, and assessment of learning diagnostic imaging.

Direct observation evaluations by emergency medicine faculty do not provide data that enhance resident assessment when compared to summative quarterly evaluations

OBJECTIVES

The purpose of this study was to compare quarterly global evaluations with direct observation evaluations to determine if direct observation evaluations provide unique data compared to those obtained from quarterly global evaluations.

METHODS

This observational, cohort study was performed at a 3-year emergency medicine (EM) residency program with 10 residents per year. Faculty used an online Web-based evaluation system to complete quarterly global evaluations and patient-specific direct observation evaluations. Two scores were collected for each resident within each quarterly evaluation period: 1) the quarterly evaluation score was the mean score across all faculty who performed a quarterly evaluation and, 2) the direct observation score was the mean score across all faculty who performed a direct observation evaluation. Pearson correlation coefficients were performed across these two groups of evaluations.

RESULTS

Over the 4-year period of the study 296 complete data sets were available for the analysis. When the quarterly evaluation score was correlated with the direct observation score for each resident at the same evaluation period, we found a very high correlation for each of the eight evaluation questions (r = 0.95-0.96, p < 0.0001). When these evaluations were stratified based on the number of direct observation evaluations that were performed during the evaluation period of interest, the correlation between the quarterly evaluation and the direct observation scores increased as the number of direct observations in the evaluation period increased. The evaluation scores from the faculty who had performed both direct observation and quarterly evaluation methods during the same resident evaluation period were highly correlated even with small numbers of evaluators.

CONCLUSIONS

Direct observations are highly correlated with quarterly evaluations when there are greater than three direct observation evaluations completed; however, this correlation drops significantly when the number of direct observations is lower. Direct observation evaluations provide similar data when compared with data obtained from quarterly global evaluations.

Short-, intermediate-, and long-term outcomes of Pennsylvania's continuum of tobacco education pilot project

INTRODUCTION

The most effective time to introduce formal tobacco use treatment training for physicians is during the medical school experience. However, few medical schools have adopted standardized curricula, missing an important opportunity to influence future physician behavior. The Pennsylvania Continuum of Tobacco Education pilot project was undertaken from spring 2003 through summer 2005 to evaluate a generalizable method of improving students' knowledge, attitudes, and behaviors related to tobacco use treatment.

METHODS

Intervention methods included a 1-day intensive multiformat seminar, followed by a reinforcement session 4 weeks later, within an internal medicine clerkship. Outcome measures included changes in students' attitudes, rates of "ask" and "advise" behaviors during clinical encounters, and performance on end-of-year clinical skills examinations.

RESULTS

Short, intermediate, and long-term outcomes related to both smoking assessment and counseling improved as a result of the intervention. The percentage of students who obtained tobacco histories and counseled patients in clerkships increased following the seminar compared with the baseline. Nearly, all students demonstrated relevant skills during a clinical skills assessment at the end of the third year.

DISCUSSION

The introduction of a standardized tobacco curriculum into medical school training is both feasible and effective. Results were sustained following the intervention, and the effects were reflected across several valid outcomes.

Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool

OBJECTIVES

A Council of Emergency Medicine Residency Directors task force developed the Standardized Direct Observation Assessment Tool (SDOT), a 26-item checklist assessment tool to evaluate Accreditation Council for Graduate Medical Education resident core competencies by direct observation. Each of the checklist items is assigned to one or more of five core competencies. The objective of this study was to test the interrater measurement properties of the SDOT instrument.

METHODS

Two videos of simulated patient-resident-attending physician encounters were produced. Academic emergency medicine faculty members not involved in the development of the form viewed the two encounters and completed the SDOT for each. Faculty demographic data were collected. Data were collected from 82 faculty members at 16 emergency medicine residency programs. The checklist items were used to generate a composite score for each core competency of patient care, medical knowledge, interpersonal and communication skills, professionalism, and systems-based practice.

RESULTS

Univariate analysis demonstrated a high degree of agreement between evaluators in evaluating residents for both videos. Multivariate analysis found no differences in rating by faculty when examined by experience, academic title, site, or previous use of the SDOT.

CONCLUSIONS

Faculty from 16 emergency medicine residency programs had a high interrater agreement when using the SDOT to evaluate resident core competency performance. This study did not test the validity of the tool. This data analysis is mainly descriptive, and scripted video scenarios may not approximate direct observation in the emergency department.

Report of the Task Force on National Fourth Year Medical Student Emergency Medicine Curriculum Guide

This manuscript reports recommendations of the National Fourth Year Medical Student Emergency Medicine Curriculum Guide Task Force. This task force was convened by 6 major emergency medicine organizations to develop a standardized curriculum for fourth year medical students. The structure of the curriculum is based on clerkship curricula from other specialties such as internal medicine and pediatrics. The report contains a historical context, global and targeted needs assessment, goals and objectives, recommended educational strategies, implementation guidelines, and suggestions on feedback and evaluation.