The assessment of individual cognitive expertise and clinical competency: a research agenda

A Literature Study of Medical Simulations for Non-Technical Skills Training in Emergency Medicine: Twenty Years of Progress, an Integrated Research Framework, and Future Research Avenues

Medical simulations have led to extensive developments in emergency medicine. Apart from the growing number of applications and research efforts in patient safety, few studies have focused on modalities, research methods, and professions via a synthesis of simulation studies with a focus on non-technical skills training. Intersections between medical simulation, non-technical skills training, and emergency medicine merit a synthesis of progress over the first two decades of the 21st century. Drawing on research from the Web of Science Core Collection's Science Citation Index Expanded and Social Science Citation Index editions, results showed that medical simulations were found to be effective, practical, and highly motivating. More importantly, simulation-based education should be a teaching approach, and many simulations are utilised to substitute high-risk, rare, and complex circumstances in technical or situational simulations. (1) Publications were grouped by specific categories of non-technical skills, teamwork, communication, diagnosis, resuscitation, airway management, anaesthesia, simulation, and medical education. (2) Although mixed-method and quantitative approaches were prominent during the time period, further exploration of qualitative data would greatly contribute to the interpretation of experience. (3) High-fidelity dummy was the most suitable instrument, but the tendency of simulators without explicitly stating the vendor selection calls for a standardised training process. The literature study concludes with a ring model as the integrated framework of presently known best practices and a broad range of underexplored research areas to be investigated in detail.

Preparing and Presenting Validation Studies: A Guide for the Perplexed

SUMMARY STATEMENT

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

School-based research agenda on healthcare simulation for nursing education in Hong Kong

Background

Healthcare simulation has been used as a pedagogical strategy in nursing education. Evidence has shown one of the positive impacts that simulations replace clinical placement. These wide-ranging initiatives are essential, and they can guide a nursing school’s simulation training. However, researching each innovation in the nursing field is beyond the scope.

Methods

To focus our research and develop the capacity and capability to incorporate healthcare simulation in nursing education, we used a consensus building process to establish a school’s research agenda. A modified Delphi process was adopted to reach a consensus among 10 nursing faculty members in one university with a visiting professor’s support.

Results

The three themes were identified as (1) embedding simulation into the baccalaureate in nursing curriculum, (2) designing effective simulation-based education and (3) simulating education in the broader world (adolescents). These themes were further categorised into two areas that used simulation in the educational and community settings. Sixty per cent of the faculty members agreed that the question, ‘How can simulation be incorporated into clinical placements to enhance students’ learning?' should be the highest research priority.

Conclusion

This study adds understanding to incorporate simulation-based education in the nursing curriculum and community provides insights into future research.

Establishing Inter- and Intrarater Reliability for High-Stakes Testing Using Simulation

AIM

This article reports one method to develop a standardized training method to establish the inter- and intrarater reliability of a group of raters for high-stakes testing.

BACKGROUND

Simulation is used increasingly for high-stakes testing, but without research into the development of inter- and intrarater reliability for raters.

METHOD

Eleven raters were trained using a standardized methodology. Raters scored 28 student videos over a six-week period. Raters then rescored all videos over a two-day period to establish both intra- and interrater reliability.

RESULTS

One rater demonstrated poor intrarater reliability; a second rater failed all students. Kappa statistics improved from the moderate to substantial agreement range with the exclusion of the two outlier raters' scores.

CONCLUSION

There may be faculty who, for different reasons, should not be included in high-stakes testing evaluations. All faculty are content experts, but not all are expert evaluators.

Beyond standard checklist assessment: Question sequence may impact student performance

INTRODUCTION

Clinical encounters are often assessed using a checklist. However, without direct faculty observation, the timing and sequence of questions are not captured. We theorized that the sequence of questions can be captured and measured using coherence scores that may distinguish between low and high performing candidates.

METHODS

A logical sequence of key features was determined using the standard case checklist for an observed structured clinical exam (OSCE). An independent clinician educator reviewed each encounter to provide a global rating. Coherence scores were calculated based on question sequence. These scores were compared with global ratings and checklist scores.

RESULTS

Coherence scores were positively correlated to checklist scores and to global ratings, and these correlations increased as global ratings improved. Coherence scores explained more of the variance in student performance as global ratings improved.

DISCUSSION

Logically structured question sequences may indicate a higher performing student, and this information is often lost when using only overall checklist scores.

CONCLUSIONS

The sequence test takers ask questions can be accurately recorded, and is correlated to checklist scores and to global ratings. The sequence of questions during a clinical encounter is not captured by traditional checklist scoring, and may represent an important dimension of performance.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents

Objective:

We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios.

Methods:

An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool.

Results:

Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada– Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada–Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho 5 [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho 5 [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p , 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFPEM residents in all scenarios (p 5 0.006 to 0.04).

Conclusions:

This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulationbased assessment tools.

Assessing patient care: summary of the breakout group on assessment of observable learner performance

There is an established expectation that physicians in training demonstrate competence in all aspects of clinical care prior to entering professional practice. Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs. A thorough assessment of competence in patient care requires a mixture of methods, taking into account each method's costs, benefits, and current level of evidence. At the 2012 Academic Emergency Medicine (AEM) consensus conference on educational research, one breakout group reviewed and discussed the evidence supporting various methods of assessing patient care and defined a research agenda for the continued development of specific assessment methods based on current best practices. In this article, the authors review each method's supporting reliability and validity evidence and make specific recommendations for future educational research.

Simulation in emergency medicine training

Simulation provides a range of educational tools that have increasingly been incorporated into emergency medicine (EM) curricula. Standardized patients and some partial task trainers, such as intubation heads, have been used for decades. More recently, a growing number of computer-screen simulations, high-fidelity mannequins, and virtual-reality simulators have expanded the number of procedures and conditions, which can be effectively simulated.The Accreditation Council for Graduate Medical Education transitioned to a competency-based assessment of residency programs in 2001 and included simulation as a method for incorporating the 6 core competencies into graduate medical education curricula. Over the past decade, numerous peer-reviewed publications have promoted simulation as an effective educational tool for each of the core competencies.The advanced technology used to operate many current simulators can erroneously become the focus of efforts to create a simulation-based curriculum. Simulation can most effectively be incorporated into EM curricula through the use of time-proven concepts, which start with defining the targeted learners, assessing their general and specific educational needs, defining learning objectives, and selecting the best educational strategy for achieving each objective. In many, but not all, instances, simulation can be the best tool for achieving EM learning objectives.

It's time: an argument for a national emergency medicine education research center

Education research in emergency medicine has made some advances, but still suffers from poorly designed studies and isolated projects that are small and cannot be generalized to other institutions. This commentary argues for the need of an emergency medicine education research group (EMERG). EMERG would facilitate and coordinate better quality educational research projects at multiple institutions. This in turn would promote faculty development in education research and potentially result in improved educational outcomes and patient care.

The use of simulation for pediatric training and assessment

PURPOSE OF REVIEW

Simulation has been widely adopted as a training and assessment tool in medical education. Conventional teaching methods may be inadequate to properly train healthcare providers for rare but potentially lethal events in pediatrics such as trauma and respiratory arrest. Recent studies suggest pediatric acute care providers have limited exposure to critically ill patients and also lack the skills to manage them. Simulation has the potential to fill this educational void. This review will highlight the role of simulation as an educational and assessment tool, with a particular emphasis on retention of knowledge and skills.

RECENT FINDINGS

Simulation is currently used as an assessment tool to provide ongoing feedback during training (formative assessment) and is gaining popularity as an adjunctive method for demonstrating competency (summative assessment). Recent literature demonstrates increased retention of knowledge and skills after simulation-based training in the areas of resuscitation, trauma, airway management, procedural training, team training, and disaster management.

SUMMARY

Simulation is an effective training tool for pediatric acute care providers. Further research is necessary to develop validated performance assessment tools and demonstrate improvement in clinical outcomes after simulation training.

Hybrid simulation combining a high fidelity scenario with a pelvic ultrasound task trainer enhances the training and evaluation of endovaginal ultrasound skills

OBJECTIVES

In this study, an endovaginal ultrasound (US) task trainer was combined with a high-fidelity US mannequin to create a hybrid simulation model. In a scenario depicting a patient with ectopic pregnancy and hemorrhagic shock, this model was compared with a standard high-fidelity simulation during training sessions with emergency medicine (EM) residents. The authors hypothesized that use of the hybrid model would increase both the residents' self-reported educational experience and the faculty's self-reported ability to evaluate the residents' skills.

METHODS

A total of 45 EM residents at two institutions were randomized into two groups. Each group was assigned to one of two formats involving an ectopic pregnancy scenario. One format incorporated the new hybrid model, in which residents had to manipulate an endovaginal US probe in a task trainer; the other used the standard high-fidelity simulation mannequin together with static photo images. After finishing the scenario, residents self-rated their overall learning experience and how well the scenario evaluated their ability to interpret endovaginal US images. Faculty members reviewed video recordings of the other institution's residents and rated their own ability to evaluate residents' skills in interpreting endovaginal US images and diagnosing and managing the case scenario. Visual analog scales (VAS) were used for the self-ratings.

RESULTS

Compared to the residents assigned to the standard simulation scenario, residents assigned to the hybrid model reported an increase in their overall educational experience (Delta VAS = 10, 95% confidence interval [CI] = 4 to 18) and felt the hybrid model was a better measure of their ability to interpret endovaginal US images (Delta VAS = 17, 95% CI = 7 to 28). Faculty members found the hybrid model to be better than the standard simulation for evaluating residents' skills in interpreting endovaginal US images (Delta VAS = 13, 95% CI = 6 to 20) and diagnosing and managing the case (Delta VAS = 10, 95% CI = 2 to 18). Time to reach a diagnosis was similar in both groups (p = 0.053).

CONCLUSIONS

Use of a hybrid simulation model combining a high-fidelity simulation with an endovaginal US task trainer improved residents' educational experience and improved faculty's ability to evaluate residents' endovaginal US and clinical skills. This novel hybrid tool should be considered for future education and evaluation of EM residents.