Performance of Residents and Anesthesiologists in a Simulation-based Skill Assessment

Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study

OBJECTIVES

The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.

DESIGN

Observational multicenter study.

SETTING

Thirteen Spanish ICU Departments.

PARTICIPANTS

Thirty six R3.

INTERVENTION

The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.

MAIN VARIABLES OF INTEREST

Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.

RESULTS

A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64-0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65-0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.

CONCLUSION

The expected level of competency after one year in the ICU was achieved only in half of the performances. A more evidence-based educational approach is needed. Multiple center simulation-based assessment showed feasibility and reliability as an evaluation method of competency.

TRIAL REGISTRATION

COBALIDATION. NCT04278976. (https://register.

CLINICALTRIALS

gov).

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Navigate the Challenges of Simulation for Assessment: A Faculty Development Workshop

INTRODUCTION

Given barriers to learner assessment in the authentic clinical environment, simulated patient encounters are gaining attention as a valuable opportunity for competency assessment across the health professions. Simulation-based assessments offer advantages over traditional methods by providing realistic clinical scenarios through which a range of technical, analytical, and communication skills can be demonstrated. However, simulation for the purpose of assessment represents a paradigm shift with unique challenges, including preservation of a safe learning environment, standardization across learners, and application of valid assessment tools. Our goal was to create an interactive workshop to equip educators with the knowledge and skills needed to conduct assessments in a simulated environment.

METHODS

Participants engaged in a 90-minute workshop with large-group facilitated discussions and small-group activities for practical skill development. Facilitators guided attendees through a simulated grading exercise followed by in-depth analysis of three types of assessment tools. Participants designed a comprehensive simulation-based assessment encounter, including selection or creation of an assessment tool.

RESULTS

We have led two iterations of this workshop, including an in-person format at an international conference and a virtual format at our institution during the COVID-19 pandemic, with a total of 93 participants. Survey responses indicated strong overall ratings and impactfulness of the workshop.

DISCUSSION

Our workshop provides a practical, evidence-based framework to guide educators in the development of a simulation-based assessment program, including optimization of the environment, design of the simulated case, and utilization of meaningful, valid assessment tools.

Anesthesia assessment in the era of competences: state of the art

Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature  shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness. Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia. Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical  framework. Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.

Serious Game for Teaching Undergraduate Medical Students in Cleft lip and Palate Treatment Protocol

OBJECTIVES

To develop and evaluate a serious game to deliver knowledge about the cleft lip with or without cleft palate (CL/P) protocol used in Craniofacial Center, Chiang Mai University.

METHODS

The game "Cleft Island" was developed then extensively evaluated using different criteria, including the measured knowledge gained by participants, and gameplay experience. A group of 53 fourth and fifth-year medical students (male: 24, female: 29) were recruited to take part in an experiment. They were from the Faculty of Medicine, Chiang Mai University, and had just rotated to the Department of Surgery. Three evaluations were conducted: 1) to test whether the players had competently gained CL/P knowledge; 2) to evaluate the usability of the game according to the System Usability Scale (SUS); and 3) to evaluate the game experience in terms of a Game Experience Questionnaire (GEQ) test.

RESULTS

The results indicate a statistically significant improvement of medical students' knowledge after performing the Wilcoxon Signed-Ranks test (p < 0.5) between pre-test and post-test scores of the same medical student group; the acceptable average SUS score (M = 55.28) of the serious game; a moderate degree of experience of the GEQ components including positive affect (M = 2.64), competence (M = 2.49), and immersion (M = 2.21).

CONCLUSIONS

Cleft Island can be used as an effective supplementary instructional material, which has the potential to provide significant knowledge of CL/P treatment protocol for the players. As far as the authors are aware, this is the first study to implement and assess a serious game for training in CL/P protocol.

Development and Validation of Content Domains for Paramedic Prehospital Performance Assessment: A Focus Group and Delphi Method Approach

BACKGROUND

The chaotic and complex nature of delivering patient care in the prehospital setting complicates the provision of real-time formative feedback to paramedic students. Although the use of simulations is widespread in emergency medical services (EMS) education, a high degree of variability precludes consistent performance assessment in EMS. Objectives: The objective of this study was to define and validate key domains required to evaluate paramedic prehospital performance. Methods: We conducted a two-phase study that combined focus group and Delphi methodology. Participants were purposefully selected to attain diverse panels regarding sex, race, ethnicity, professional roles, levels of education, geographical area, and experience as a paramedic and educator. In Phase I, a panel of 11 subject matter experts (SMEs) were tasked with identifying the essential domains to be evaluated in a paramedic performance assessment. In Phase II, another panel of 11 SMEs and a four-round modified Delphi method with 28 paramedic program directors were used to validate the domains identified in Phase I. Results: The first focus group identified and achieved consensus on five domains: (1) effective communication, (2) scene management, (3) patient assessment, (4) patient management, and (5) professional behavior. These domains were validated by the second focus group. The first round of the Delphi process generated 64 content domains, which were reduced to nine unique content domains via thematic analysis. These nine content domains fit well within the broader domains identified by the focus groups with one specific area, critical thinking and reasoning, being listed in two key areas based on the definitions of the focus group domains of patient assessment and patient management. Conclusion: The content domains identified in this study provide EMS educators a theoretical framework for designing the performance assessment of newly trained professionals in the prehospital setting.

When the Learner Is the Expert: A Simulation-Based Curriculum for Emergency Medicine Faculty

Emergency physicians supervise residents performing rare clinical procedures, but they infrequently perform those procedures independently. Simulation offers a forum to practice procedural skills, but simulation labs often target resident learners, and barriers exist to faculty as learners in simulation-based training. Simulation-based curricula focused on improving emergency medicine (EM) faculty’s rare procedure skills were not discovered on review of published literature. Our objective was to create a sustainable, simulation-based faculty education curriculum for rare procedural skills in EM. Between 2012 and 2019, most EM teaching faculty at a single, urban, Level 1 trauma center completed an annual two-hour simulation-based rare procedure lab with small-group learning and guided hands-on instruction, covering 30 different procedural education sessions for faculty learners. A questionnaire administered before and after each session assessed EM faculty physicians’ self-perceived ability to perform these rare procedures. Participants’ self-reported confidence in their performance improved for all procedures, regardless of prior procedural experience. Faculty participation was initially mandatory, but is now voluntary. Diverse strategies were used to address barriers in this learner group including eliciting learner feedback, offering continuing medical education credits, gradual roll-out of checklist assessments, and welcoming expertise of faculty leaders from EM and other specialties and professions. Participants perceived training to be most helpful for the most rarely-encountered clinical procedures. Similar curricula could be implemented with minimal risk at other institutions.

Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework

Introduction

Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments.

Methods

Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors.

Results

We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts.

Conclusion

Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Protocol for the Effectiveness of an Anesthesiology Control Tower System in Improving Perioperative Quality Metrics and Clinical Outcomes: the TECTONICS randomized, pragmatic trial

Introduction: Perioperative morbidity is a public health priority, and surgical volume is increasing rapidly. With advances in technology, there is an opportunity to research the utility of a telemedicine-based control center for anesthesia clinicians that assess risk, diagnoses negative patient trajectories, and implements evidence-based practices. Objectives: The primary objective of this trial is to determine whether an anesthesiology control tower (ACT) prevents clinically relevant adverse postoperative outcomes including 30-day mortality, delirium, respiratory failure, and acute kidney injury. Secondary objectives are to determine whether the ACT improves perioperative quality of care metrics including management of temperature, mean arterial pressure, mean airway pressure with mechanical ventilation, blood glucose, anesthetic concentration, antibiotic redosing, and efficient fresh gas flow. Methods and analysis: We are conducting a single center, randomized, controlled, phase 3 pragmatic clinical trial. A total of 58 operating rooms are randomized daily to receive support from the ACT or not. All adults (eighteen years and older) undergoing surgical procedures in these operating rooms are included and followed until 30 days after their surgery. Clinicians in operating rooms randomized to ACT support receive decision support from clinicians in the ACT. In operating rooms randomized to no intervention, the current standard of anesthesia care is delivered. The intention-to-treat principle will be followed for all analyses. Differences between groups will be presented with 99% confidence intervals; p-values <0.005 will be reported as providing compelling evidence, and p-values between 0.05 and 0.005 will be reported as providing suggestive evidence. Registration: TECTONICS is registered on ClinicalTrials.gov, NCT03923699; registered on 23 April 2019.

Simulation-based assessment in anesthesia: an international multicentre validation study

PURPOSE

Simulated clinical events provide a means to evaluate a practitioner's performance in a standardized manner for all candidates that are tested. We sought to provide evidence for the validity of simulation-based assessment tools in simulated pediatric anesthesia emergencies.

METHODS

Nine centres in two countries recruited subjects to participate in simulated operating room events. Participants ranged in anesthesia experience from junior residents to staff anesthesiologists. Performances were video recorded for review and scored by specially trained, blinded, expert raters. The rating tools consisted of scenario-specific checklists and a global rating scale that allowed the rater to make a judgement about the subject's performance, and by extension, preparedness for independent practice. The reliability of the tools was classified as "substantial" (intraclass correlation coefficients ranged from 0.84 to 0.96 for the checklists and from 0.85 to 0.94 for the global rating scale).

RESULTS

Three-hundred and ninety-one simulation encounters were analysed. Senior trainees and staff significantly out-performed junior trainees (P = 0.04 and P < 0.001 respectively). The effect size of grade (junior vs senior trainee vs staff) on performance was classified as "medium" (partial η² = 0.06). Performance deficits were observed across all grades of anesthesiologist, particularly in two of the scenarios.

CONCLUSIONS

This study supports the validity of our simulation-based anesthesiologist assessment tools in several domains of validity. We also describe some residual challenges regarding the validity of our tools, some notes of caution in terms of the intended consequences of their use, and identify opportunities for further research.

Simulation-Based Assessment of Critical Care "Front-Line" Providers

OBJECTIVES

Develop a standardized simulation method to assess clinical skills of ICU providers.

DESIGN

Simulation assessment.

SETTING

Simulation laboratory.

SUBJECTS

Residents, Critical Care Medicine fellows, acute care nurse practitioner students.

INTERVENTIONS

Performance scoring in scenarios from multiple Critical Care Medicine competency domains.

MEASUREMENTS AND MAIN RESULTS

Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively.

CONCLUSIONS

The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.

Simulation Training for Residents Focused on Mechanical Ventilation: A Randomized Trial Using Mannequin-Based Versus Computer-Based Simulation

Supplemental digital content is available in the text.

Introduction

Advances in knowledge regarding mechanical ventilation (MV), in particular lung-protective ventilation strategies, have been shown to reduce mortality. However, the translation of these advances in knowledge into better therapeutic performance in real-life clinical settings continues to lag. High-fidelity simulation with a mannequin allows students to interact in lifelike situations; this may be a valuable addition to traditional didactic teaching. The purpose of this study is to compare computer-based and mannequin-based approaches for training residents on MV.

Methods

This prospective randomized single-blind trial involved 50 residents. All participants attended the same didactic lecture on respiratory pathophysiology and were subsequently randomized into two groups: the mannequin group (n = 25) and the computer screen–based simulator group (n = 25). One week later, each underwent a training assessment using five different scenarios of acute respiratory failure of different etiologies. Later, both groups underwent further testing of patient management, using in situ high-fidelity simulation of a patient with acute respiratory distress syndrome.

Results

Baseline knowledge was not significantly different between the two groups (P = 0.72). Regarding the training assessment, no significant differences were detected between the groups. In the final assessment, the scores of only the mannequin group significantly improved between the training and final session in terms of either global rating score [3.0 (2.5–4.0) vs. 2.0 (2.0–3.0), P = 0.005] or percentage of key score (82% vs. 71%, P = 0.001).

Conclusions

Mannequin-based simulation has the potential to improve skills in managing MV.

The Impact of Telemedicine on Teamwork and Workload in Pediatric Resuscitation: A Simulation-Based, Randomized Controlled Study

BACKGROUND

Telemedicine provides access to specialty care to critically ill patients from a geographic distance. The effects of using telemedicine on (1) teamwork and communication (TC), (2) task workload during resuscitation, and (3) the processes of critical care have not been well described.

OBJECTIVES

To evaluate the impact of telemedicine on (1) TC, (2) task workload during a resuscitation, and (3) the processes of critical care during a simulated pediatric resuscitation.

METHODS

Prospective single-center randomized trial. Teams of two physicians (senior and junior resident) and two standardized confederate nurses were randomized to either telemedicine (telepresent senior physician team leader) or usual care (both physicians in the room) during a simulated infant resuscitation. Simulations were video recorded and assessed for teamwork, workload, and processes of care using the Simulated Team Assessment Tool (STAT), the NASA Task Load Index (NASA-TLX) tool, and time between onset of ventricular fibrillation and defibrillation, respectively.

RESULTS

Twenty teams participated. There was no difference in teamwork between the groups (mean STAT score 72% vs. 69%; p = 0.383); however, there was a significantly greater workload in the telemedicine group (mean TLX score 56% vs. 48%, p = 0.020). Using linear regression, no difference was found in time-to-defibrillation between groups (p = 0.671), but higher teamwork scores predicted faster time to defibrillation (p = 0.020).

CONCLUSIONS

In this simulation-based study, a telepresent team leader was associated with increased team workload compared to usual care. However, no differences were noted in teamwork and processes of care metrics.

Acquiring skills in malignant hyperthermia crisis management: comparison of high-fidelity simulation versus computer-based case study

Introduction

The primary purpose of this study was to compare the effect of high fidelity simulation versus a computer-based case solving self-study, in skills acquisition about malignant hyperthermia on first year anesthesiology residents.

Methods

After institutional ethical committee approval, 31 first year anesthesiology residents were enrolled in this prospective randomized single-blinded study. Participants were randomized to either a High Fidelity Simulation Scenario or a computer-based Case Study about malignant hyperthermia. After the intervention, all subjects’ performance in was assessed through a high fidelity simulation scenario using a previously validated assessment rubric. Additionally, knowledge tests and a satisfaction survey were applied. Finally, a semi-structured interview was done to assess self-perception of reasoning process and decision-making.

Results

28 first year residents finished successfully the study. Resident's management skill scores were globally higher in High Fidelity Simulation versus Case Study, however they were significant in 4 of the 8 performance rubric elements: recognize signs and symptoms (p = 0.025), prioritization of initial actions of management (p = 0.003), recognize complications (p = 0.025) and communication (p = 0.025). Average scores from pre- and post-test knowledge questionnaires improved from 74% to 85% in the High Fidelity Simulation group, and decreased from 78% to 75% in the Case Study group (p = 0.032). Regarding the qualitative analysis, there was no difference in factors influencing the student's process of reasoning and decision-making with both teaching strategies.

Conclusion

Simulation-based training with a malignant hyperthermia high-fidelity scenario was superior to computer-based case study, improving knowledge and skills in malignant hyperthermia crisis management, with a very good satisfaction level in anesthesia residents.

Evaluation of a Simpler Tool to Assess Nontechnical Skills During Simulated Critical Events

INTRODUCTION

Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale.

METHODS

Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall.

RESULTS

The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74.

CONCLUSIONS

Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes

Background

Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.

Methods

Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.

Results

Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P &lt; 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.

Conclusions

Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Randomized controlled trials of simulation-based interventions in Emergency Medicine: a methodological review

The number of trials assessing Simulation-Based Medical Education (SBME) interventions has rapidly expanded. Many studies show that potential flaws in design, conduct and reporting of randomized controlled trials (RCTs) can bias their results. We conducted a methodological review of RCTs assessing a SBME in Emergency Medicine (EM) and examined their methodological characteristics. We searched MEDLINE via PubMed for RCT that assessed a simulation intervention in EM, published in 6 general and internal medicine and in the top 10 EM journals. The Cochrane Collaboration risk of Bias tool was used to assess risk of bias, intervention reporting was evaluated based on the "template for intervention description and replication" checklist, and methodological quality was evaluated by the Medical Education Research Study Quality Instrument. Reports selection and data extraction was done by 2 independents researchers. From 1394 RCTs screened, 68 trials assessed a SBME intervention. They represent one quarter of our sample. Cardiopulmonary resuscitation (CPR) is the most frequent topic (81%). Random sequence generation and allocation concealment were performed correctly in 66 and 49% of trials. Blinding of participants and assessors was performed correctly in 19 and 68%. Risk of attrition bias was low in three-quarters of the studies (n = 51). Risk of selective reporting bias was unclear in nearly all studies. The mean MERQSI score was of 13.4/18.4% of the reports provided a description allowing the intervention replication. Trials assessing simulation represent one quarter of RCTs in EM. Their quality remains unclear, and reproducing the interventions appears challenging due to reporting issues.

Simulation for Assessment of Milestones in Emergency Medicine Residents

OBJECTIVES

All residency programs in the United States are required to report their residents' progress on the milestones to the Accreditation Council for Graduate Medical Education (ACGME) biannually. Since the development and institution of this competency-based assessment framework, residency programs have been attempting to ascertain the best ways to assess resident performance on these metrics. Simulation was recommended by the ACGME as one method of assessment for many of the milestone subcompetencies. We developed three simulation scenarios with scenario-specific milestone-based assessment tools. We aimed to gather validity evidence for this tool.

METHODS

We conducted a prospective observational study to investigate the validity evidence for three mannequin-based simulation scenarios for assessing individual residents on emergency medicine (EM) milestones. The subcompetencies (i.e., patient care [PC]1, PC2, PC3) included were identified via a modified Delphi technique using a group of experienced EM simulationists. The scenario-specific checklist (CL) items were designed based on the individual milestone items within each EM subcompetency chosen for assessment and reviewed by experienced EM simulationists. Two independent live raters who were EM faculty at the respective study sites scored each scenario following brief rater training. The inter-rater reliability (IRR) of the assessment tool was determined by measuring intraclass correlation coefficient (ICC) for the sum of the CL items as well as the global rating scales (GRSs) for each scenario. Comparing GRS and CL scores between various postgraduate year (PGY) levels was performed with analysis of variance.

RESULTS

Eight subcompetencies were chosen to assess with three simulation cases, using 118 subjects. Evidence of test content, internal structure, response process, and relations with other variables were found. The ICCs for the sum of the CL items and the GRSs were >0.8 for all cases, with one exception (clinical management GRS = 0.74 in sepsis case). The sum of CL items and GRSs (p < 0.05) discriminated between PGY levels on all cases. However, when the specific CL items were mapped back to milestones in various proficiency levels, the milestones in the higher proficiency levels (level 3 [L3] and 4 [L4]) did not often discriminate between various PGY levels. L3 milestone items discriminated between PGY levels on five of 12 occasions they were assessed, and L4 items discriminated only two of 12 times they were assessed.

CONCLUSION

Three simulation cases with scenario-specific assessment tools allowed evaluation of EM residents on proficiency L1 to L4 within eight of the EM milestone subcompetencies. Evidence of test content, internal structure, response process, and relations with other variables were found. Good to excellent IRR and the ability to discriminate between various PGY levels was found for both the sum of CL items and the GRSs. However, there was a lack of a positive relationship between advancing PGY level and the completion of higher-level milestone items (L3 and L4).

The Utility of Simulation in the Management of Patients With Congenital Heart Disease: Past, Present, and Future

Significant advancements have been made in the diagnosis and management of congenital heart disease (CHD). As a result, a higher percentage of these patients are surviving to adulthood. Despite this improvement in management, these patients remain at higher risk of morbidity and mortality, particularly in the perioperative setting. One new area of interest in these patients is the implementation of simulation-based medical education. Simulation has demonstrated various benefits across high-acuity scenarios encountered in the hospital. In CHD, simulation has been used in the training of pediatrics residents, assessment of intraoperative complications, echocardiography, and anatomic modeling with 3-dimensional printing. Here, we describe the current state of simulation in CHD, its role in training care providers for the management of this population, and future directions of CHD simulation.

Simulation and the diagnostic process: a pilot study of trauma and rapid response teams

Background:

Simulation is frequently used to recreate many of the crises encountered in patient care settings. Teams learn to manage these crises in an environment that maximizes their learning experiences and eliminates the potential for patient harm. By designing simulation scenarios that include conditions associated with diagnostic errors, teams can experience how their decisions can lead to errors. The purpose of this study was to assess how trauma teams (TrT) and pediatric rapid response teams (RRT) managed scenarios that included a diagnostic error.

Methods:

We developed four scenarios that would require TrT and pediatric RRT to manage an error in diagnosis. The two trauma scenarios (spinal cord injury and tracheobronchial tear) were designed to not respond to the heuristic management approach frequently used in trauma settings. The two pediatric scenarios (foreign body aspiration and coarctation of the aorta) had an incorrect diagnosis on admission. Two raters independently scored the scenarios using a rating system based on how teams managed the diagnostic process (search, establish and confirm a new diagnosis and initiate therapy based on the new diagnosis).

Results:

Twenty-one TrT and 17 pediatric rapid response managed 51 scenarios. All of the teams questioned the initial diagnosis. The teams were able to establish and confirm a new diagnosis in 49% of the scenarios (25 of 51). Only 23 (45%) teams changed their management of the patient based on the new diagnosis.

Conclusions:

Simulation can be used to recreate conditions that engage teams in the diagnostic process. In contrast to most instruction about diagnostic error, teams learn through realistic experiences and receive timely feedback about their decision-making skills. Based on the findings in this pilot study, the majority of teams would benefit from an education intervention designed to improve their diagnostic skills.

Rescuing the Clinical Breast Examination: Advances in Classifying Technique and Assessing Physician Competency

OBJECTIVE

Develop new performance evaluation standards for the clinical breast examination (CBE).

SUMMARY BACKGROUND DATA

There are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy.

METHODS

This performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy.

RESULTS

Accuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001).

CONCLUSIONS

Our results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

Avoiding premature closure and reaching diagnostic accuracy: some key predictive factors

CONTEXT

Early studies suggested that experienced clinicians simply generate more accurate diagnoses than those less experienced. However, more recent studies indicate that experienced clinicians may be subject to biases in formulating and confirming hypotheses that lead to inaccuracy.

OBJECTIVES

The goal of this study was to identify factors associated with the ability to process information in ways that overcome premature closure and result in accurate diagnosis using a set of vignettes in which inconsistent information was introduced midway.

METHODS

Seventy-five participants (25 Year 3 medical students, 25 internal medicine residents in their second year of residency and 25 internal medicine faculty) were recruited to solve each of four complex clinical vignettes. In each vignette, the first four rounds of information pointed toward a narrowing range of diagnostic possibilities, but patient information presented in and after the fifth round was inconsistent with prior findings. In addition to accuracy, outcome measures were length of differential diagnosis, certainty of diagnosis, persistence in data collection and tendency to switch diagnoses.

RESULTS

There were no significant differences in diagnostic accuracy across the three groups, each of which differed in level of training. However, across experience levels, diagnostic accuracy was associated with the mean number of items in the differential, tendency to persist (e.g. to request a greater number of rounds of information), and openness to switch (e.g. to change the most likely diagnosis on receipt of disconfirming information).

CONCLUSIONS

Level of training (i.e. clinical experience) was not associated with accuracy on this task. As faculty clinicians certainly have more knowledge than their junior counterparts, it is important to identify ways in which cognitive factors can lead to more or less persistence and openness, and to teach clinicians how to overcome tendencies associated with error.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists

Background

We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods.

Methods

A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant’s technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist.

Results

Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance.

Conclusions

Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Physiological and self-assessed psychological stress induced by a high fidelity simulation course among third year anesthesia and critical care residents: An observational study

INTRODUCTION

The use of high fidelity simulators in Medicine can improve knowledge, behaviour and practice but may be associated with significant stress. Our objective was to measure physiological and psychological self-assessed intensity of stress before and after a planned simulation training session among third year anaesthesia and critical care residents.

METHODS

A convenience sample of 27 residents participating in a simulation training course was studied. Stress was evaluated by self-assessment using a numerical scale and by salivary amylase concentration before and after the session. Technical and non-technical (using the Aberdeen Anaesthetists' Non Technical Skills scale) performances were assessed through videotapes analysis.

RESULTS

The median stress score was 5 (2-8) before and 7 (2-10) after the simulation session (P<0.001). For 48% of residents studied, the stress score after the session was superior or equal to 8/10. Salivary amylase concentration increased significantly after the session compared to before the session, respectively (1,250,440±1,216,667 vs. 727,260±603,787IU/L, P=0.008). There was no significant correlation between stress parameters and non-technical performance.

DISCUSSION

Simulation-induced stress, as measured by self-assessment and biological parameter, is high before the session and increases significantly during the course. While this stress did not seem to impact performance negatively, it should be taken into account.

Effect of a simulation-based workshop on breaking bad news for anesthesiology residents: an intervention study

Background

Breaking bad news (BBN) to patients and their relatives is a complex and stressful task. The ideal structure, training methods and assessment instruments best used to teach and assess BBN for anesthesiology residents remain unclear. The purpose of this study is to evaluate the effectiveness of an education intervention for BBN based on immersive experiences with a high fidelity simulator and role-play with standardized patients (SPs). A secondary purpose is to gather validity evidence to support the use of a GRIEV_ING instrument to assess BBN skills.

Methods

The communication skills for BBN of 16 residents were assessed via videotaped SP encounters at baseline and immediately post-intervention. Residents’ perceptions about their ability and comfort for BBN were collected using pre and post workshop surveys.

Results

Posttest scores were significantly higher than the pretest scores for the GRIEV_ING checklist, as well as on the communication global rating. The GRIEV_ING checklist had acceptable inter-rater and internal-consistency reliabilities. Performance was not related to years of training, or previous BBN experience.

Conclusion

Anesthesiology residents’ communication skills when BBN in relation to a critical incident may be improved with educational interventions based on immersive experiences with a high fidelity simulator and role-play with SPs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12871-017-0374-7) contains supplementary material, which is available to authorized users.

Performance gaps and improvement plans from a 5-hospital simulation programme for anaesthesiology providers: a retrospective study

Background

Simulation is increasingly employed in healthcare provider education, but usage as a means of identifying system-wide practitioner gaps has been limited. We sought to determine whether practice gaps could be identified, and if meaningful improvement plans could result from a simulation course for anaesthesiology providers.

Methods

Over a 2-year cycle, 288 anaesthesiologists and 67 certified registered nurse anaesthetists (CRNAs) participated in a 3.5 hour, malpractice insurer-mandated simulation course, encountering 4 scenarios. 5 anaesthesiology departments within 3 urban academic healthcare systems were represented. A real-time rater scored each individual on 12 critical performance items (CPIs) representing learning objectives for a given scenario. Participants completed a course satisfaction survey, a 1-month postcourse practice improvement plan (PIP) and a 6-month follow-up survey.

Results

All recorded course data were retrospectively reviewed. Course satisfaction was generally positive (88–97% positive rating by item). 4231 individual CPIs were recorded (of a possible 4260 rateable), with a majority of participants demonstrating remediable gaps in medical/technical and non-technical skills (97% of groups had at least one instance of a remediable gap in communication/non-technical skills during at least one of the scenarios). 6 months following the course, 91% of respondents reported successfully implementing 1 or more of their PIPs. Improvements in equipment/environmental resources or personal knowledge domains were most often successful, and several individual reports demonstrated a positive impact on actual practice.

Conclusions

This professional liability insurer-initiated simulation course for 5 anaesthesiology departments was feasible to deliver and well received. Practice gaps were identified during the course and remediation of gaps, and/or application of new knowledge, skills and resources was reported by participants.

Reliability and validity of a tool to assess airway management skills in anesthesia trainees

BACKGROUND AND AIMS

Gaining expertise in procedural skills is essential for achieving clinical competence during anesthesia training. Supervisors have the important responsibility of deciding when the trainee can be allowed to perform various procedures without direct supervision while ensuring patient safety. This requires robust and reliable assessment techniques. Airway management with bag-mask ventilation and tracheal intubation are routinely performed by anesthesia trainees at induction of anesthesia and to save lives during a cardiorespiratory arrest. The purpose of this study was to evaluate the construct validity, and inter-rater and test-retest reliability of a tool designed to assess competence in bag-mask ventilation followed by tracheal intubation in anesthesia trainees.

MATERIAL AND METHODS

Informed consent was obtained from all participants. Tracheal intubation and bag-mask ventilation skills in 10 junior and 10 senior anesthesia trainees were assessed by two investigators on two occasions at a 3-4 weeks interval, using a procedure-specific assessment tool.

RESULTS

Average kappa value for inter-rater reliability was 0.91 and 0.99 for the first and second assessments, respectively, with an average agreement of 95%. The average agreement for test-retest reliability was 82% with a kappa value of 0.39. Senior trainees obtained higher scores compared to junior trainees in all areas of assessment, with a significant difference for patient positioning, preoxygenation, and laryngoscopy technique, depicting good construct validity.

CONCLUSION

The tool designed to assess bag-mask ventilation and tracheal intubation skills in anesthesia trainees demonstrated excellent inter-rater reliability, fair test-retest reliability, and good construct validity. The authors recommend its use for formative and summative assessment of junior anesthesia trainees.

A Smartphone-based Decision Support Tool Improves Test Performance Concerning Application of the Guidelines for Managing Regional Anesthesia in the Patient Receiving Antithrombotic or Thrombolytic Therapy

BACKGROUND

The American Society of Regional Anesthesia and Pain Medicine (ASRA) consensus statement on regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy is the standard for evaluation and management of these patients. The authors hypothesized that an electronic decision support tool (eDST) would improve test performance compared with native physician behavior concerning the application of this guideline.

METHODS

Anesthesiology trainees and faculty at 8 institutions participated in a prospective, randomized trial in which they completed a 20-question test involving clinical scenarios related to the ASRA guidelines. The eDST group completed the test using an iOS app programmed to contain decision logic and content of the ASRA guidelines. The control group completed the test by using any resource in addition to the app. A generalized linear mixed-effects model was used to examine the effect of the intervention.

RESULTS

After obtaining institutional review board's approval and informed consent, 259 participants were enrolled and randomized (eDST = 122; control = 137). The mean score was 92.4 ± 6.6% in the eDST group and 68.0 ± 15.8% in the control group (P < 0.001). eDST use increased the odds of selecting correct answers (7.8; 95% CI, 5.7 to 10.7). Most control group participants (63%) used some cognitive aid during the test, and they scored higher than those who tested from memory alone (76 ± 15% vs. 57 ± 18%, P < 0.001). There was no difference in time to completion of the test (P = 0.15) and no effect of training level (P = 0.56).

CONCLUSIONS

eDST use improved application of the ASRA guidelines compared with the native clinician behavior in a testing environment.

Simulation-based assessments in health professional education: a systematic review

INTRODUCTION

The use of simulation in health professional education has increased rapidly over the past 2 decades. While simulation has predominantly been used to train health professionals and students for a variety of clinically related situations, there is an increasing trend to use simulation as an assessment tool, especially for the development of technical-based skills required during clinical practice. However, there is a lack of evidence about the effectiveness of using simulation for the assessment of competency. Therefore, the aim of this systematic review was to examine simulation as an assessment tool of technical skills across health professional education.

METHODS

A systematic review of Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (ERIC), Medical Literature Analysis and Retrieval System Online (Medline), and Web of Science databases was used to identify research studies published in English between 2000 and 2015 reporting on measures of validity, reliability, or feasibility of simulation as an assessment tool. The McMasters Critical Review for quantitative studies was used to determine methodological value on all full-text reviewed articles. Simulation techniques using human patient simulators, standardized patients, task trainers, and virtual reality were included.

RESULTS

A total of 1,064 articles were identified using search criteria, and 67 full-text articles were screened for eligibility. Twenty-one articles were included in the final review. The findings indicated that simulation was more robust when used as an assessment in combination with other assessment tools and when more than one simulation scenario was used. Limitations of the research papers included small participant numbers, poor methodological quality, and predominance of studies from medicine, which preclude any definite conclusions.

CONCLUSION

Simulation has now been embedded across a range of health professional education and it appears that simulation-based assessments can be used effectively. However, the effectiveness as a stand-alone assessment tool requires further research.

Checklists and multidisciplinary team performance during simulated obstetric hemorrhage

BACKGROUND

Checklists can optimize team performance during medical crises. However, there has been limited examination of checklist use during obstetric crises. In this simulation study we exposed multidisciplinary teams to checklist training to evaluate checklist use and team performance during a severe postpartum hemorrhage.

METHODS

Fourteen multidisciplinary teams participated in a postpartum hemorrhage simulation occurring after vaginal delivery. Before participating, each team received checklist training. The primary study outcome was whether each team used the checklist during the simulation. Secondary outcomes were the times taken to activate our institution-specific massive transfusion protocol and commence red blood cell transfusion, and whether a designated checklist reader was used.

RESULTS

The majority of teams (12/14 (86%)) used the checklist. Red blood cell transfusion was administered by all teams. The median [IQR] times taken to activate the massive transfusion protocol and transfuse red blood cells were 5min 14s [3:23-6:43] and 14min 40s [12:56-17:28], respectively. A designated checklist reader was used by 7/12 (58%) teams that used the checklist. Among teams that used a checklist with versus without a designated reader, we observed no differences in the times to activate the massive transfusion protocol or to commence red blood cell transfusion (P>0.05).

CONCLUSIONS

Although checklist training was effective in promoting checklist use, multidisciplinary teams varied in their scope of checklist use during a postpartum hemorrhage simulation. Future studies are required to determine whether structured checklist training can result in more standardized checklist use during a postpartum hemorrhage.

Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program

BACKGROUND

This study describes anesthesiologists' practice improvements undertaken during the first 3 yr of simulation activities for the Maintenance of Certification in Anesthesiology Program.

METHODS

A stratified sampling of 3 yr (2010-2012) of participants' practice improvement plans was coded, categorized, and analyzed.

RESULTS

Using the sampling scheme, 634 of 1,275 participants in Maintenance of Certification in Anesthesiology Program simulation courses were evaluated from the following practice settings: 41% (262) academic, 54% (339) community, and 5% (33) military/other. A total of 1,982 plans were analyzed for completion, target audience, and topic. On follow-up, 79% (1,558) were fully completed, 16% (310) were partially completed, and 6% (114) were not completed within the 90-day reporting period. Plans targeted the reporting individual (89% of plans) and others (78% of plans): anesthesia providers (50%), non-anesthesia physicians (16%), and non-anesthesia non-physician providers (26%). From the plans, 2,453 improvements were categorized as work environment or systems changes (33% of improvements), teamwork skills (30%), personal knowledge (29%), handoff (4%), procedural skills (3%), or patient communication (1%). The median word count was 63 (interquartile range, 30 to 126) for each participant's combined plans and 147 (interquartile range, 52 to 257) for improvement follow-up reports.

CONCLUSIONS

After making a commitment to change, 94% of anesthesiologists participating in a Maintenance of Certification in Anesthesiology Program simulation course successfully implemented some or all of their planned practice improvements. This compares favorably to rates in other studies. Simulation experiences stimulate active learning and motivate personal and collaborative practice improvement changes. Further evaluation will assess the impact of the improvements and further refine the program.

Every surgical resident should know how to perform a cricothyrotomy: an inexpensive cricothyrotomy task trainer for teaching and assessing surgical trainees

OBJECTIVE

Emergency cricothyrotomy is a rare but potentially lifesaving procedure. Training opportunities for surgical residents to learn this skill are limited, and many graduating residents have never performed one during their training. We aimed to develop and validate a novel and inexpensive cricothyrotomy task trainer that can be constructed from household items.

DESIGN

A model was constructed using a toilet paper roll (trachea and larynx), Styrofoam (soft tissue), cardboard (thyroid cartilage), zip tie (cricoid), and fabric (skin). Participants were asked to complete a simulated cricothyrotomy procedure using the model. They were then evaluated using a 10-point checklist (5 points total) devised by 6 general surgeons. Participants were also asked to complete an anonymous survey rating the educational value and the degree of enjoyment regarding the model.

SETTING

A tertiary care teaching hospital.

PARTICIPANTS

A total of 54 students and general surgery residents (11 medical students, 32 interns, and 11 postgraduate year 3 residents).

RESULTS

All 54 participants completed the training and assessment. The scores ranged from 0 to 5. The mean (range) scores were 1.8 (1-4) for medical students, 3.5 (1-5) for junior residents, and 4.9 (4-5) for senior-level residents. Medical students were significantly outperformed by junior- and senior-level residents (p < 0.001). Trainees felt that the model was educational (4.5) and enjoyable (4.0).

CONCLUSIONS

A low-fidelity, low-cost cricothyrotomy simulator distinguished the performance of emergency cricothyrotomy between medical students and junior- and senior-level general surgery residents. This task trainer may be ideally suited to providing basic skills to all physicians in training, especially in settings with limited resources and clinical opportunities.

Simulation as a new tool to establish benchmark outcome measures in obstetrics

BACKGROUND

There are not enough clinical data from rare critical events to calculate statistics to decide if the management of actual events might be below what could reasonably be expected (i.e. was an outlier).

OBJECTIVES

In this project we used simulation to describe the distribution of management times as an approach to decide if the management of a simulated obstetrical crisis scenario could be considered an outlier.

DESIGN

Twelve obstetrical teams managed 4 scenarios that were previously developed. Relevant outcome variables were defined by expert consensus. The distribution of the response times from the teams who performed the respective intervention was graphically displayed and median and quartiles calculated using rank order statistics.

RESULTS

Only 7 of the 12 teams performed chest compressions during the arrest following the 'cannot intubate/cannot ventilate' scenario. All other outcome measures were performed by at least 11 of the 12 teams. Calculation of medians and quartiles with 95% CI was possible for all outcomes. Confidence intervals, given the small sample size, were large.

CONCLUSION

We demonstrated the use of simulation to calculate quantiles for management times of critical event. This approach could assist in deciding if a given performance could be considered normal and also point to aspects of care that seem to pose particular challenges as evidenced by a large number of teams not performing the expected maneuver. However sufficiently large sample sizes (i.e. from a national data base) will be required to calculate acceptable confidence intervals and to establish actual tolerance limits.

Validity and reliability assessment of detailed scoring checklists for use during perioperative emergency simulation training

INTRODUCTION

Few valid and reliable grading checklists have been published for the evaluation of performance during simulated high-stakes perioperative event management. As such, the purposes of this study were to construct valid scoring checklists for a variety of perioperative emergencies and to determine the reliability of scores produced by these checklists during continuous video review.

METHODS

A group of anesthesiologists, intensivists, and educators created a set of simulation grading checklists for the assessment of the following scenarios: severe anaphylaxis, cerebrovascular accident, hyperkalemic arrest, malignant hyperthermia, and acute coronary syndrome. Checklist items were coded as critical or noncritical. Nonexpert raters evaluated 10 simulation videos in a random order, with each video being graded 4 times. A group of faculty experts also graded the videos to create a reference standard to which nonexpert ratings were compared. P < 0.05 was considered significant.

RESULTS

Team leaders in the simulation videos were scored by the expert panel as having performed 56.5% of all items on the checklist (range, 43.8%-84.0%), and 67.2% of the critical items (range, 30.0%-100%). Nonexpert raters agreed with the expert assessment 89.6% of the time (95% confidence interval, 87.2%-91.6%). No learning curve development was found with repetitive video assessment or checklist use. The κ values comparing nonexpert rater assessments to the reference standard averaged 0.76 (95% confidence interval, 0.71-0.81).

CONCLUSIONS

The findings indicate that the grading checklists described are valid, are reliable, and could be used in perioperative crisis management assessment.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents

Objective:

We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios.

Methods:

An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool.

Results:

Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada– Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada–Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho 5 [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho 5 [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p , 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFPEM residents in all scenarios (p 5 0.006 to 0.04).

Conclusions:

This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulationbased assessment tools.

The role of simulation in continuing medical education for acute care physicians: a systematic review

OBJECTIVES

We systematically reviewed the effectiveness of simulation-based education, targeting independently practicing qualified physicians in acute care specialties. We also describe how simulation is used for performance assessment in this population.

DATA SOURCES

Data source included: DataMEDLINE, Embase, Cochrane Database of Systematic Reviews, Cochrane CENTRAL Database of Controlled Trials, and National Health Service Economic Evaluation Database. The last date of search was January 31, 2013.

STUDY SELECTION

All original research describing simulation-based education for independently practicing physicians in anesthesiology, critical care, and emergency medicine was reviewed.

DATA EXTRACTION

Data analysis was performed in duplicate with further review by a third author in cases of disagreement until consensus was reached. Data extraction was focused on effectiveness according to Kirkpatrick's model. For simulation-based performance assessment, tool characteristics and sources of validity evidence were also collated.

DATA SYNTHESIS

Of 39 studies identified, 30 studies focused on the effectiveness of simulation-based education and nine studies evaluated the validity of simulation-based assessment. Thirteen studies (30%) targeted the lower levels of Kirkpatrick's hierarchy with reliance on self-reporting. Simulation was unanimously described as a positive learning experience with perceived impact on clinical practice. Of the 17 remaining studies, 10 used a single group or "no intervention comparison group" design. The majority (n = 17; 44%) were able to demonstrate both immediate and sustained improvements in educational outcomes. Nine studies reported the psychometric properties of simulation-based performance assessment as their sole objective. These predominantly recruited independent practitioners as a convenience sample to establish whether the tool could discriminate between experienced and inexperienced operators and concentrated on a single aspect of validity evidence.

CONCLUSIONS

Simulation is perceived as a positive learning experience with limited evidence to support improved learning. Future research should focus on the optimal modality and frequency of exposure, quality of assessment tools and on the impact of simulation-based education beyond the individuals toward improved patient care.

The impact of exposure to liver transplantation anesthesia on the ability to treat intraoperative hyperkalemia: a simulation experience

The objective of this study was to assess whether resident exposure to liver transplantation anesthesia results in improved patient care during a simulated critical care scenario. Our hypothesis was that anesthesia residents exposed to liver transplantation anesthesia care would be able to identify and treat a simulated hyperkalemic crisis after reperfusion more appropriately than residents who have not been involved in liver transplantation anesthesia care. Participation in liver transplantation anesthesia is not a mandatory component of the curriculum of anesthesiology training programs in the United States. It is unclear whether exposure to liver transplantation anesthesia is beneficial for skill set development. A high-fidelity human patient simulation scenario was developed. Times for administration of epinephrine, calcium chloride, and secondary hyperkalemia treatment were recorded. A total of 25 residents with similar training levels participated: 13 residents had previous liver transplantation experience (OLT), whereas 12 residents had not been previously exposed to liver transplantations (non-OLT). The OLT group performed better in recognizing and treating the hyperkalemic crisis than the non-OLT group. Pharmacologic therapy for hyperkalemia was given earlier (OLT 53.3 ± 27.0 seconds versus non-OLT 148 ± 104.1 seconds; P < 0.01) and hemodynamics restored quicker (OLT 87.9 ± 24.9 seconds versus non-OLT 219.9 ± 87.1 seconds; P < 0.01). Simulation-based assessment of clinical skills is a useful tool for evaluating anesthesia resident performance during an intraoperative crisis situation related to liver transplantations. Previous liver transplantation experience improves the anesthesia resident's ability to recognize and treat hyperkalemic cardiac arrest.

Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated "Reader"

BACKGROUND AND OBJECTIVES

A hardcopy or paper cognitive aid has been shown to improve performance during the management of simulated local anesthetic systemic toxicity (LAST) when given to the team leader. However, there remains room for improvement to ensure a system that can achieve perfect adherence to the published guidelines for LAST management. Recent research has shown that implementing a checklist via a designated reader may be of benefit. Accordingly, we sought to investigate the effect of an electronic decision support tool (DST) and designated "Reader" role on team performance during an in situ simulation of LAST.

METHODS

Participants were randomized to Reader + DST (n = 16, rDST) and Control (n = 15, memory alone). The rDST group received the assistance of a dedicated Reader on the response team who was equipped with an electronic DST. The primary outcome measure was adherence to guidelines.

RESULTS

For overall and critical percent correct scores, the rDST group scored higher than Control (99.3% vs 72.2%, P < 0.0001; 99.5% vs 70%, P < 0.0001, respectively). In the LAST scenario, 0 (0%) of 15 in the control group performed 100% of critical management steps, whereas 15 (93.8%) of 16 in the rDST group did so (P < 0.0001).

CONCLUSIONS

In a prospective, randomized single-blinded study, a designated Reader with an electronic DST improved adherence to guidelines in the management of an in situ simulation of LAST. Such tools are promising in the future of medicine, but further research is needed to ensure the best methods for implementing them in the clinical arena.