Development of an Objective Scoring System for Measurement of Resident Performance on the Human Patient Simulator

Validating Parallel-Forms Tests for Assessing Anesthesia Resident Knowledge

We created a serious game to teach first year anesthesiology (CA-1) residents to perform general anesthesia for cesarean delivery. We aimed to investigate resident knowledge gains after playing the game and having received one of 2 modalities of debriefing. We report on the development and validation of scores from parallel test forms for criterion-referenced interpretations of resident knowledge. The test forms were intended for use as pre- and posttests for the experiment. Validation of instruments measuring the study's primary outcome was considered essential for adding rigor to the planned experiment, to be able to trust the study's results. Parallel, multiple-choice test forms development steps included: (1) assessment purpose and population specification; (2) content domain specification and writing/selection of items; (3) content validation by experts of paired items by topic and cognitive level; and (4) empirical validation of scores from the parallel test forms using Classical Test Theory (CTT) techniques. Field testing involved online administration of 52 shuffled items from both test forms to 24 CA-1's, 21 second-year anesthesiology (CA-2) residents, 2 fellows, 1 attending anesthesiologist, and 1 of unknown rank at 3 US institutions. Items from each form yielded near-normal score distributions, with similar medians, ranges, and standard deviations. Evaluations of CTT item difficulty (item p values) and discrimination (D) indices indicated that most items met assumptions of criterion-referenced test design, separating experienced from novice residents. Experienced residents performed better on overall domain scores than novices (P < .05). Kuder-Richardson Formula 20 (KR-20) reliability estimates of both test forms were above the acceptability cut of .70, and parallel forms reliability estimate was high at .86, indicating results were consistent with theoretical expectations. Total scores of parallel test forms demonstrated item-level validity, strong internal consistency and parallel forms reliability, suggesting sufficient robustness for knowledge outcomes assessments of CA-1 residents.

Automating medical simulations

OBJECTIVE

This study aims to explore speech as an alternative modality for human activity recognition (HAR) in medical settings. While current HAR technologies rely on video and sensory modalities, they are often unsuitable for the medical environment due to interference from medical personnel, privacy concerns, and environmental limitations. Therefore, we propose an end-to-end, fully automatic objective checklist validation framework that utilizes medical personnel's uttered speech to recognize and document the executed actions in a checklist format.

METHODS

Our framework records, processes, and analyzes medical personnel's speech to extract valuable information about performed actions. This information is then used to fill the corresponding rubrics in the checklist automatically.

RESULTS

Our approach to activity recognition outperformed the online expert examiner, achieving an F1 score of 0.869 on verbal tasks and an ICC score of 0.822 with an offline examiner. Furthermore, the framework successfully identified communication failures and medical errors made by physicians and nurses.

CONCLUSION

Implementing a speech-based framework in medical settings, such as the emergency room and operation room, holds promise for improving care delivery and enabling the development of automated assistive technologies in various medical domains. By leveraging speech as a modality for HAR, we can overcome the limitations of existing technologies and enhance workflow efficiency and patient safety.

Global obstetric anaesthesia: bridging the gap in maternal health care inequities through partnership in education

Maternal mortality rates are unacceptably high globally. Low- and middle-income countries (LMICs) face challenges of an inadequate anaesthesia workforce, under-resourced healthcare systems and sub-optimal access to labour and delivery care, all of which negatively impact maternal and neonatal outcomes. In order to effect the changes in surgical-obstetric-anaesthesia workforce numbers advocated by the Lancet Commission on Global Surgery to support the UN sustainable development goals, mass training and upskilling of both physician and non-physician anaesthetists is imperative. The implementation of outreach programmes and partnerships across organisations and countries has already been shown to improve the provision of safe care to mothers and their babies, and these efforts should be continued. Short subspecialty courses and simulation training are two cornerstones of modern obstetric anaesthesia training in poorly resourced environments. This review discusses the challenges to accessing quality maternal healthcare in LMICs and the use of education, outreach, partnership and research to protect the most vulnerable women from coming to harm in the peripartum period.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Updates on Simulation in Obstetrical Anesthesiology Through the COVID-19 Pandemic

Simulation has played a critical role in medicine for decades as a pedagogical and assessment tool. The labor and delivery unit provides an ideal setting for the use of simulation technology. Prior reviews of this topic have focused on simulation for individual and team training and assessment. The COVID-19 pandemic has provided an opportunity for educators and leaders in obstetric anesthesiology to rapidly train health care providers and develop new protocols for patient care with simulation. This review surveys new developments in simulation for obstetric anesthesiology with an emphasis on simulation use during the COVID-19 pandemic.

Simulation-Based Assessment Identifies Longitudinal Changes in Cognitive Skills in an Anesthesiology Residency Training Program

OBJECTIVES

We describe observed improvements in nontechnical or "higher-order" deficiencies and cognitive performance skills in an anesthesia residency cohort for a 1-year time interval. Our main objectives were to evaluate higher-order, cognitive performance and to demonstrate that simulation can effectively serve as an assessment of cognitive skills and can help detect "higher-order" deficiencies, which are not as well identified through more traditional assessment tools. We hypothesized that simulation can identify longitudinal changes in cognitive skills and that cognitive performance deficiencies can then be remediated over time.

METHODS

We used 50 scenarios evaluating 35 residents during 2 subsequent years, and 18 of those 35 residents were evaluated in both years (post graduate years 3 then 4) in the same or similar scenarios. Individual basic knowledge and cognitive performance during simulation-based scenarios were assessed using a 20- to 27-item scenario-specific checklist. Items were labeled as basic knowledge/technical (lower-order cognition) or advanced cognitive/nontechnical (higher-order cognition). Identical or similar scenarios were repeated annually by a subset of 18 residents during 2 successive academic years. For every scenario and item, we calculated group error scenario rate (frequency) and individual (resident) item success. Grouped individuals' success rates are calculated as mean (SD), and item success grade and group error rates are calculated and presented as proportions. For all analyses, α level is 0.05.

RESULTS

Overall PGY4 residents' error rates were lower and success rates higher for the cognitive items compared with technical item performance in the operating room and resuscitation domains. In all 3 clinical domains, the cognitive error rate by PGY4 residents was fairly low (0.00-0.22) and the cognitive success rate by PGY4 residents was high (0.83-1.00) and significantly better compared with previous annual assessments (P < 0.05). Overall, there was an annual decrease in error rates for 2 years, primarily driven by decreases in cognitive errors. The most commonly observed cognitive error types remained anchoring, availability bias, premature closure, and confirmation bias.

CONCLUSIONS

Simulation-based assessments can highlight cognitive performance areas of relative strength, weakness, and progress in a resident or resident cohort. We believe that they can therefore be used to inform curriculum development including activities that require higher-level cognitive processing.

General anesthesia for emergency cesarean delivery: simulation-based evaluation of residents

Introduction

Decreased rates of General Anesthesia (GA) for Cesarean Section (C-section) create a learning problem for anesthesia trainees. In this context, training the management of GA for C-section using simulation techniques allows a safe environment for exposure, learning, performance improvement, and capability retention.

Objective

Analyze anesthesia residents’ performance regarding a simulated clinical case of GA for emergency C-section and identify specific deficits in skill acquisition.

Methods

Between 2015 and 2018, we evaluated the performance of 25 anesthesiology residents challenged by a simulated clinical case of GA for emergency C-section after the conclusion of the obstetric anesthesia rotation. Each resident performed the clinical case once followed by the assessment of their performance. Final scores were given according to the completion rate of 14-tasks, going from 0% to 100%. Two study groups were considered according to residency year for subsequent comparison of results (Group 1, second and third residency years and Group 2, fourth and fifth residency years).

Results and discussion

Mean score was 64.29% ± 13.62. Comparatively, Group 1 obtained a higher score than Group 2 (70.63% ± 14.02 vs. 60.27% ± 11.94), although with no statistically significant difference (p = 0.063). The tasks most frequently accomplished were opioid administration (100%), rapid sequence technique (100%), pre-oxygenation (92%), gastric content aspiration prophylaxis (84%), and previous clinical history (84%). Conversely, the tasks less frequently accomplished were confirming presence of pediatrician (64%), oxytocin administration (56%), PONV prophylaxis (56%), and preoperative airway assessment (48%).

Conclusion

The performance of the residents observed in this study was comparable to results previously published. The final score did not depend on the residency year.

Impact of repeated simulation on learning curve characteristics of residents exposed to rare life threatening situations

Background

Little is known about the learning curve characteristics of residents undertaking simulation-based education. It is important to understand the time for acquisition and decay of knowledge and skills needed to manage rare and difficult clinical situations.

Method

Ten anaesthesiology residents underwent simulation-based education to manage a cannot intubate cannot ventilate scenario during general anaesthesia for caesarean section. Their performance was measured using an assessment tool and debriefed by two experienced anaesthesiologists. The parameters against which the performance was judged were grouped into preoperative assessment, preoperative patient care, equipment availability, induction sequence, communication and adherence to airway algorithm protocol. The scenario was repeated at 6 and 12 months thereafter. The residents’ acquisition of knowledge, technical and non-technical skills were assessed and compared at baseline, 6 months and end of 12 months.

Result

The skills of preoperative assessment, preoperative care and communication quickly improved but the specific skill of managing a difficult airway as measured by adherence to an airway algorithm required more than 6 months (CI at 6 vs 12 months: −3.4 to –0.81, p=0.016). The skills of preoperative assessment and preoperative care improved to a higher level quickly and were retained at this improved level. Communication (CI at 0 vs 6 months: −3.78 to −0.22, p=0.045 and at 6 vs 12 months : −3.39 to −1.49, p=0.007) and difficult airway management skill were slower to improve but continued to do so over the 12 months. The compliance to machine check was more gradual and showed an improvement at 12 months.

Conclusion

Our study is unique in analysing the learning curve characteristics of different components of a failed obstetric airway management skill. Repeated simulations over a longer period of time help in better reinforcement, retention of knowledge, recapitulation and implementation of technical and non-technical skills.

The Process of Developing an Assessment Checklist for Simulated Infant Respiratory Distress Using a Modified Delphi Method: A Mixed Methods Study

Introduction

Assessing clinical performance, such as managing respiratory distress, in clinical trainees is challenging yet important. Our objective was to describe and evaluate an integrative and iterative approach to developing a checklist measuring simulated clinical performance for infant respiratory distress.

Methods

We implemented a five-step modified Delphi process with an embedded qualitative component. An implementation period occurred followed by a second qualitative data collection. Validity evidence was collected throughout the process.

Results

A 19-item assessment checklist was developed for managing infant respiratory distress by medical student learners in a simulation-based setting. The iterative process provided content validity while the qualitative data provided response process validity. Cohen kappa was 0.82 indicating strong rater agreement. The assessment checklist was found to be easy to use and measure what was intended.

Conclusion

We developed an accurate and reliable assessment checklist for medical student learners in a simulation-based learning setting with high interrater reliability and validity evidence. Given its ease of use, we encourage medical educators and researchers to utilize this method to develop and implement assessment checklists for their interventions.

Randomized Controlled Simulation Trial to Compare Transfer Procedures for Emergency Cesarean

OBJECTIVE

To test the hypothesis that capping intravenous and epidural lines would reduce time to transfer women in labor to the operating room and time to readiness for general anesthesia for emergency cesarean. The secondary purpose was to identify latent threats to patient safety.

DESIGN

Mixed methods analysis of a randomized, controlled, in situ simulation trial.

SETTING

Labor and delivery unit at high-risk referral center.

PARTICIPANTS

Fifteen interprofessional teams that included labor and delivery nurses and anesthesiology residents.

METHODS

Immediately before simulation, we randomized bedside nurses and anesthesiology residents to one of two groups: usual transfer or the cap and run procedure. Simulation scenarios started with fetal heart rate decelerations that necessitated position changes followed by emergency cesarean. An embedded simulated obstetrician announced the decision for cesarean; completion of an OR checklist confirmed team readiness to induce general anesthesia. Postsimulation debriefing was focused on teamwork and opportunities to improve safety, and we used qualitative analysis to synthesize results.

RESULTS

We found no statistically significant difference in the overall time from decision for cesarean to readiness for general anesthesia between the two groups (usual transfer median = 445 seconds [interquartile range, 425-465] vs. cap and run 390 seconds [interquartile range, 383-443], p = .12). The time in the operating room was less in the cap and run group than in the usual transfer group (median = 300 seconds vs. 250 seconds, p = .038). Qualitative analysis of the debriefing data indicated advantages of the capping procedure, including better bed maneuverability and fewer tangled lines.

CONCLUSION

We found no evidence of decreased overall time from decision for cesarean to readiness for general anesthesia based on whether the nurse capped the intravenous and epidural lines or pushed the intravenous pole alongside the bed. However, nurses perceived improved patient safety with the cap and run procedure.

Obstetric Anesthesia: Leading the Way in Patient Safety

The subspecialty of obstetric anesthesiology has embraced patient safety research, which has led to a reduction in obstetric anesthesia-related morbidity and mortality. Although there are innumerable individual improvements, this article highlights the following innovations: safer and more effective labor analgesia, safer treatments for hypotension associated with neuraxial blockade, advances in spinal and epidural techniques for operative deliveries, lower incidence of postdural puncture headache through improved technology, safer parental agents for labor analgesia, improved safety of general anesthesia in obstetrics, improved education and the use of simulation including team training, and reductions in operating room-related infections.

Recent advances of simulation in obstetric anesthesia

PURPOSE OF REVIEW

Simulation training in obstetric anesthesia has become widespread in recent years. Simulations are used to train staff and trainees, assess and improve team performance, and evaluate the work environment. This review summarizes current research in these categories.

RECENT FINDINGS

Simulation to improve individual technical skills has focused on induction of general anesthesia for emergent cesarean delivery, an infrequently encountered scenario by anesthesia trainees. Low- and high-fidelity simulation devices for the learning and practicing neuraxial and non-neuraxial procedures have been described, and both are equally effective. The use of checklists in obstetric emergencies has become common as and post-scenario debriefing techniques have improved. Although participant task performance improves, whether participants retain learned skills or whether simulation improves patient outcomes has not yet been established. Tools to assess teamwork during simulation have been developed, but none have been rigorously validated. In-situ vs. offsite simulations do not differ in effectiveness.

SUMMARY

Simulation allows for practice of tasks and teamwork in a controlled manner. There is little data whether simulation improves patient outcomes and metrics to predict the long-term retention of skills by simulation participants have not been developed.

Simulation for Assessment of Milestones in Emergency Medicine Residents

OBJECTIVES

All residency programs in the United States are required to report their residents' progress on the milestones to the Accreditation Council for Graduate Medical Education (ACGME) biannually. Since the development and institution of this competency-based assessment framework, residency programs have been attempting to ascertain the best ways to assess resident performance on these metrics. Simulation was recommended by the ACGME as one method of assessment for many of the milestone subcompetencies. We developed three simulation scenarios with scenario-specific milestone-based assessment tools. We aimed to gather validity evidence for this tool.

METHODS

We conducted a prospective observational study to investigate the validity evidence for three mannequin-based simulation scenarios for assessing individual residents on emergency medicine (EM) milestones. The subcompetencies (i.e., patient care [PC]1, PC2, PC3) included were identified via a modified Delphi technique using a group of experienced EM simulationists. The scenario-specific checklist (CL) items were designed based on the individual milestone items within each EM subcompetency chosen for assessment and reviewed by experienced EM simulationists. Two independent live raters who were EM faculty at the respective study sites scored each scenario following brief rater training. The inter-rater reliability (IRR) of the assessment tool was determined by measuring intraclass correlation coefficient (ICC) for the sum of the CL items as well as the global rating scales (GRSs) for each scenario. Comparing GRS and CL scores between various postgraduate year (PGY) levels was performed with analysis of variance.

RESULTS

Eight subcompetencies were chosen to assess with three simulation cases, using 118 subjects. Evidence of test content, internal structure, response process, and relations with other variables were found. The ICCs for the sum of the CL items and the GRSs were >0.8 for all cases, with one exception (clinical management GRS = 0.74 in sepsis case). The sum of CL items and GRSs (p < 0.05) discriminated between PGY levels on all cases. However, when the specific CL items were mapped back to milestones in various proficiency levels, the milestones in the higher proficiency levels (level 3 [L3] and 4 [L4]) did not often discriminate between various PGY levels. L3 milestone items discriminated between PGY levels on five of 12 occasions they were assessed, and L4 items discriminated only two of 12 times they were assessed.

CONCLUSION

Three simulation cases with scenario-specific assessment tools allowed evaluation of EM residents on proficiency L1 to L4 within eight of the EM milestone subcompetencies. Evidence of test content, internal structure, response process, and relations with other variables were found. Good to excellent IRR and the ability to discriminate between various PGY levels was found for both the sum of CL items and the GRSs. However, there was a lack of a positive relationship between advancing PGY level and the completion of higher-level milestone items (L3 and L4).

Improving Patient Safety through Simulation Training in Anesthesiology: Where Are We?

There have been colossal technological advances in the use of simulation in anesthesiology in the past 2 decades. Over the years, the use of simulation has gone from low fidelity to high fidelity models that mimic human responses in a startlingly realistic manner, extremely life-like mannequin that breathes, generates E.K.G, and has pulses, heart sounds, and an airway that can be programmed for different degrees of obstruction. Simulation in anesthesiology is no longer a research fascination but an integral part of resident education and one of ACGME requirements for resident graduation. Simulation training has been objectively shown to increase the skill-set of anesthesiologists. Anesthesiology is leading the movement in patient safety. It is rational to assume a relationship between simulation training and patient safety. Nevertheless there has not been a demonstrable improvement in patient outcomes with simulation training. Larger prospective studies that evaluate the improvement in patient outcomes are needed to justify the integration of simulation training in resident education but ample number of studies in the past 5 years do show a definite benefit of using simulation in anesthesiology training. This paper gives a brief overview of the history and evolution of use of simulation in anesthesiology and highlights some of the more recent studies that have advanced simulation-based training.

Development and content validation of performance assessments for endoscopic third ventriculostomy

PURPOSE

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

METHODS

The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.

RESULTS

Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80% agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50-80% agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.

CONCLUSIONS

We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

Operationalising elaboration theory for simulation instruction design: a Delphi study

OBJECTIVE

The aim of this study was to assess the feasibility of incorporating the Delphi process within the simplifying conditions method (SCM) described in elaboration theory (ET) to identify conditions impacting the complexity of procedural skills for novice learners.

METHODS

We generated an initial list of conditions impacting the complexity of lumbar puncture (LP) from key informant interviews (n = 5) and a literature review. Eighteen clinician-educators from six different medical specialties were subsequently recruited as expert panellists. Over three Delphi rounds, these panellists rated: (i) their agreement with the inclusion of the simple version of the conditions in a representative ('epitome') training scenario, and (ii) how much the inverse (complex) version increases LP complexity for a novice. Cronbach's α-values were used to assess inter-rater agreement.

RESULTS

All panellists completed Rounds 1 and 2 of the survey and 17 completed Round 3. In Round 1, Cronbach's α-values were 0.89 and 0.94 for conditions that simplify and increase LP complexity, respectively; both values increased to 0.98 in Rounds 2 and 3. With the exception of 'high CSF (cerebral spinal fluid) pressure', panellists agreed with the inclusion of all conditions in the simplest (epitome) training scenario. Panellists rated patient movement, spinal anatomy, patient cooperativeness, body habitus, and the presence or absence of an experienced assistant as having the greatest impact on the complexity of LP.

CONCLUSIONS

This study demonstrated the feasibility of using expert consensus to establish conditions impacting the complexity of procedural skills, and the benefits of incorporating the Delphi method into the SCM. These data can be used to develop and sequence simulation scenarios in a progressively challenging manner. If the theorised learning gains associated with ET are realised, the methods described in this study may be applied to the design of simulation training for other procedural and non-procedural skills, thereby advancing the agenda of theoretically based instruction design in health care simulation.

A modified Delphi method to create a scoring system for assessing team performance during maternal cardiopulmonary arrest

BACKGROUND

Maternal cardiopulmonary arrest is a rare but often fatal emergency. The authors used a modified Delphi method to create a checklist of tasks for practitioners.

METHODS

Within each round, experts ranked tasks on a scale from zero through five. Consensus was defined a priori as 80% exact agreement.

RESULTS

Three rounds were required to achieve consensus resulting in a checklist of 45 tasks. Round One results revealed five tasks, Round Two included 25 tasks, and Round Three resulted in 29 tasks with 80% exact agreement.

CONCLUSIONS

The modified Delphi method resulted in a weighted scoring system that can be used to objectively assess team performance.

Exploring patterns of error in acute care using framework analysis

BACKGROUND

Junior doctors are often the first responders to deteriorating patients in hospital. In the high-stakes and time-pressured context of acute care, the propensity for error is high. This study aimed to identify the main subject areas in which junior doctors' acute care errors occur, and cross-reference the errors with Reason's Generic Error Modelling System (GEMS). GEMS categorises errors according to the underlying cognitive processes, and thus provides insight into the causative factors. The overall aim of this study was to identify patterns in junior doctors' acute care errors in order to enhance understanding and guide the development of educational strategies.

METHODS

This observational study utilised simulated acute care scenarios involving junior doctors dealing with a range of emergencies. Scenarios and the subsequent debriefs were video-recorded. Framework analysis was used to categorise the errors according to eight inductively-developed key subject areas. Subsequently, a multi-dimensional analysis was performed which cross-referenced the key subject areas with an earlier categorisation of the same errors using GEMS. The numbers of errors in each category were used to identify patterns of error.

RESULTS

Eight key subject areas were identified; hospital systems, prioritisation, treatment, ethical principles, procedural skills, communication, situation awareness and infection control. There was a predominance of rule-based mistakes in relation to the key subject areas of hospital systems, prioritisation, treatment and ethical principles. In contrast, procedural skills, communication and situation awareness were more closely associated with skill-based slips and lapses. Knowledge-based mistakes were less frequent but occurred in relation to hospital systems and procedural skills.

CONCLUSIONS

In order to improve the management of acutely unwell patients by junior doctors, medical educators must understand the causes of common errors. Adequate knowledge alone does not ensure prompt and appropriate management and referral. The teaching of acute care skills may be enhanced by encouraging medical educators to consider the range of potential error types, and their relationships to particular tasks and subjects. Rule-based mistakes may be amenable to simulation-based training, whereas skill-based slips and lapses may be reduced using strategies designed to raise awareness of the interplay between emotion, cognition and behaviour.

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

BACKGROUND

It is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program.

OBJECTIVE

We assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination.

METHODS

Fifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items).

RESULTS

The OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents.

CONCLUSIONS

The comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.

Five steps to develop checklists for evaluating clinical performance: an integrative approach

PURPOSE

The process of developing checklists to rate clinical performance is essential for ensuring their quality; thus, the authors applied an integrative approach for designing checklists that evaluate clinical performance.

METHOD

The approach consisted of five predefined steps (taken 2012-2013). Step 1: On the basis of the relevant literature and their clinical experience, the authors drafted a preliminary checklist. Step 2: The authors sent the draft checklist to five experts who reviewed it using an adapted Delphi technique. Step 3: The authors devised three scoring categories for items after pilot testing. Step 4: To ensure the changes made after pilot testing were valid, the checklist was submitted to an additional Delphi review round. Step 5: To weight items needed for accurate performance assessment, 10 pediatricians rated all checklist items in terms of their importance on a scale from 1 (not important) to 5 (essential).

RESULTS

The authors have illustrated their approach using the example of a checklist for a simulation scenario of infant septic shock. The five-step approach resulted in a valid, reliable tool and proved to be an effective method to design evaluation checklists. It resulted in 33 items, most consisting of three scoring categories.

CONCLUSIONS

This approach integrates published evidence and the knowledge of domain experts. A robust development process is a necessary prerequisite of valid performance checklists. Establishing a widely recognized standard for developing evaluation checklists will likely support the design of appropriate measurement tools and move the field of performance assessment in health care forward.

Simulation-based assessment to evaluate cognitive performance in an anesthesiology residency program

BACKGROUND

Problem solving in a clinical context requires knowledge and experience, and most traditional examinations for learners do not capture skills that are required in some situations where there is uncertainty about the proper course of action.

OBJECTIVE

We sought to evaluate anesthesiology residents for deficiencies in cognitive performance within and across 3 clinical domains (operating room, trauma, and cardiac resuscitation) using simulation-based assessment.

METHODS

Individual basic knowledge and cognitive performance in each simulation-based scenario were assessed in 47 residents using a 15- to 29-item scenario-specific checklist. For every scenario and item we calculated group error scenario rate (frequency) and individual (resident) item success. For all analyses, alpha was designated as 0.05.

RESULTS

Postgraduate year (PGY)-3 and PGY-4 residents' cognitive items error rates were higher and success rates lower compared to basic and technical performance in each domain tested (P < .05). In the trauma and resuscitation scenarios, the cognitive error rate by PGY-4 residents was fairly high (0.29-0.5) and their cognitive success rate was low (0.5-0.68). The most common cognitive errors were anchoring, availability bias, premature closure, and confirmation bias.

CONCLUSIONS

Simulation-based assessment can differentiate between higher-order (cognitive) and lower-order (basic and technical) skills expected of relatively experienced (PGY-3 and PGY-4) anesthesiology residents. Simulation-based assessments can also highlight areas of relative strength and weakness in a resident group, and this information can be used to guide curricular modifications to address deficiencies in tasks requiring higher-order processing and cognition.

Recent trends in simulation for obstetric anesthesia

PURPOSE OF REVIEW

The use of simulation in medicine has grown significantly over the past 2 decades. Simulation in obstetric anesthesia can be divided into four broad uses: technical skills, nontechnical or teamwork skills, individual clinical competence, and the safety of the clinical environment. This review will describe recent trends in the use of simulation in several of these categories.

RECENT FINDINGS

Simulation continues to be an important part of skills (technical and cognitive) and teamwork training in obstetric anesthesia. The acquisition of simple and complex technical skills appears to be improved with the use of simulation. However, the assessment of these skills is currently done in the simulated environment. Simulation is also important in assessing and enhancing the safety of a labor unit. Two simulation trends have recently evolved. Instructional articles describing how to best perform simulation have begun to appear. In addition, several review articles have been published that demonstrate the maturation of the body of research in this field.

SUMMARY

As the use of simulation continues to grow, research should concentrate on whether anesthesia or teamwork skills learned in the simulated environment change behavior and improve outcomes in the clinical setting. More instructional publications would also facilitate the growth into more clinical environments.

Evaluation of an integrated graphical display to promote acute change detection in ICU patients

OBJECTIVE

The purpose of this study was to evaluate ICU nurses' ability to detect patient change using an integrated graphical information display (IGID) versus a conventional tabular ICU patient information display (i.e. electronic chart).

DESIGN

Using participants from two different sites, we conducted a repeated measures simulator-based experiment to assess ICU nurses' ability to detect abnormal patient variables using a novel IGID versus a conventional tabular information display. Patient scenarios and display presentations were fully counterbalanced.

MEASUREMENTS

We measured percent correct detection of abnormal patient variables, nurses' perceived workload (NASA-TLX), and display usability ratings.

RESULTS

32 ICU nurses (87% female, median age of 29 years, and median ICU experience of 2.5 years) using the IGID detected more abnormal variables compared to the tabular display [F(1, 119)=13.0, p<0.05]. There was a significant main effect of site [F(1, 119)=14.2], with development site participants doing better. There were no significant differences in nurses' perceived workload. The IGID display was rated as more usable than the conventional display [F(1, 60)=31.7].

CONCLUSION

Overall, nurses reported more important physiological information with the novel IGID than tabular display. Moreover, the finding of site differences may reflect local influences in work practice and involvement in iterative display design methodology. Information displays developed using user-centered design should accommodate the full diversity of the intended user population across use sites.

A tool for immediate and automated assessment of resuscitation skills for a full-scale simulator

Background

For performance assessment during simulation, mostly observers rate the trainees' performance using checklists. Simulator outcome may provide immediate and objective feedback to the participants but requires additional work for the accurate scenario design. High-fidelity simulators are based on physiologic models and store all changes of the simulator conditions during the scenarios and may therefore be used for the assessment of performance. In the present work, the design of a simulator script for the assessment of resuscitation skills using an Emergency Care Simulator (ECS, METI, Sarasota, Florida) is described.

Findings

A standardized resuscitation simulator script and a visual basic-based macro were programmed for the immediate and automated extraction of performance-related variables from the log files. The following parameters were assessed: mean cardiac output, time until return of spontaneous circulation, no-flow-time, no-flow-time fraction, the time until the first defibrillation, the number and fraction of indicated and non-indicated defibrillations. Furthermore, mean deviation of defibrillation interval from the 2 minutes interval, the mean interval of defibrillations and the time until the first administration of epinephrine were calculated. As an example, the results of resuscitation efforts according to 2005 guidelines by five teams that consisted of one emergency physician and two paramedics are presented. No data are provided about its validity and reliability.

Conclusion

The tool can be used to assess adherence to European and American cardiopulmonary resuscitation guidelines (both 2005 and 2010) and to compare simulator outcome if different guidelines are trained and applied according to specific curricula. It represents an example of how simulator outcome can be used for performance assessment and may help to design more complex test-scenarios including the field of critical incidents in anesthesia.

Short simulation training improves objective skills in established advanced practitioners managing emergencies on the ward and surgical intensive care unit

BACKGROUND

Several studies evaluating simulation training in intensive care unit (ICU) physicians have demonstrated improvement in leadership and management skills. No study to date has evaluated whether such training is useful in established ICU advanced practitioners (APs). We hypothesized that human patient simulator-based training would improve surgical ICU APs' skills at managing medical crises.

METHODS

After institutional review board approval, 12 APs completed ½ day of simulation training on the SimMan, Laerdal system. Each subject participated in five scenarios, first as team leader (pretraining scenario), then as observer for three scenarios, and finally, again as team leader (posttraining). Faculty teaching accompanied each scenario and preceded a debriefing session with video replay. Three experts scored emergency care skills (Airway-Breathing-Circulation [ABCs] sequence, recognition of shock, pneumothorax, etc.) and teamwork leadership/interpersonal skills. A multiple choice question examination and training effectiveness questionnaire were completed before and after training. Fellows underwent the same curriculum and served to validate the study. Pre- and postscores were compared using the Wilcoxon signed rank test with two-tailed significance of 0.05.

RESULTS

Improvement was seen in participants' scores combining all parameters (73% ± 13% vs. 80% ± 11%, p = 0.018). AP leadership/interpersonal skills (+12%), multiple choice question examination (+4%), and training effectiveness questionnaire (+6%) scores improved significantly (p < 0.05). Fellows teamwork leadership/interpersonal skills scores were higher than APs (p < 0.001) but training brought AP scores to fellow levels. Interrater reliability was high (r = 0.77, 95% confidence interval 0.71-0.82; p < 0.001).

CONCLUSIONS

Human patient simulator training in established surgical ICU APs improves leadership, teamwork, and self-confidence skills in managing medical emergencies. Such a validated curriculum may be useful as an AP continuing education resource.

Defining awakening from anesthesia in neonates: a consensus study

BACKGROUND

A descriptive tool for determining awakening in infants is desirable to test the value of depth of anesthesia monitors. Although scales and criteria have been developed for children and infants, none has been applied to the study of anesthetised neonates. We aimed to seek consensus in a group of experts on a definition of awakening at the end of anesthesia in neonates.

METHODS

We used a modified Delphi technique with an iterative process of questionnaires and anonymised feedback. Communication was conducted by email. Thirty-one consultant pediatric anesthetists in the UK and Ireland took part. Consensus was defined a priori as 80% agreement.

RESULTS

The 83% of respondents agreed that defining awakening is possible. Consensus was reached on six criteria and also that a combination of these criteria must be used. As crying and attempting to cry are similar, we propose that at least two of the following five behaviors are present to consider a neonate awake after anesthesia: (i) crying or attempting to cry, (ii) vigorous limb movements, (iii) gagging on a tracheal tube, (iv) eyes open, and (v) looking around. There was also consensus that three stimuli are appropriate to test rousability in neonates awakening from anesthesia: (i) removal of skin adhesive tape, (ii) stroking/tickling the skin or gentle shaking, and (iii) pharyngeal suction.

CONCLUSIONS

We propose a scale for determining awakening from anesthesia in neonates that may be used in future studies, particularly regarding electroencephalographic data and depth of anesthesia monitoring in neonates.

Medical simulation in respiratory and critical care medicine

Simulation-based medical education has gained tremendous popularity over the past two decades. Driven by the patient safety movement, changes in the educational opportunities available to trainees and the rapidly evolving capabilities of computer technology, simulation-based medical education is now being used across the continuum of medical education. This review provides the reader with a perspective on simulation specific to respiratory and critical care medicine, including an overview of historical and modern simulation modalities and the current evidence supporting their use.

The Functions of Team Monitoring and ‘Talking to the Room’ for Performance in Anesthesia Teams

We explored the functions of team monitoring and talking to the room (undirected talk to relay relevant information or comment on the performance of real-time self behavior) for effects on team interaction and performance in 27 anesthesia teams in the clinical setting. Immediate reactions to team monitoring and talking to the room were investigated by means of lag sequential analysis. As expected, we found that in high performing teams, immediate consequences of team monitoring were speaking-up and providing unsolicited assistance. Talking to the room led to further talking to the room and substituted explicit coordination. The results highlight the relevance of team monitoring and talking to the room for team coordination and performance in dynamic healthcare environments.

Managing nonroutine events in anesthesia: the role of adaptive coordination

OBJECTIVE

This field study aimed at examining the role of anesthesia teams' adaptive coordination in managing changing situational demands, such as in nonroutine events (NREs).

BACKGROUND

Medical teams' ability to adapt their teamwork (e.g., their coordination activities) to changing situational demands is crucial to team performance and, thus, to patient safety. Whereas the majority of previous studies on the matter have focused on critical but rare events, it has recently been pointed out that the effective management of NREs is a key challenge to medical teams. Hence this study investigated the relationship between coordination activities, NRE occurrence, and team performance.

METHOD

We videotaped 22 anesthesia teams during standard anesthesia induction and recorded data from the vital signs monitor and the ventilator. Coordination was coded by a trained observer using a structured observation system. NREs were recorded by an experienced staff anesthesiologist using all three video streams. Checklist-based team performance assessment was also performed by an experienced staff anesthesiologist.

RESULTS

We found that anesthesia teams adapt their coordination activities to changing situational demands. In particular, the increased occurrence of NREs caused an increase in the time the teams spent on task management. A stronger increase in the teams' task management (i.e., more adaptive coordination) was related to their performance.

CONCLUSION

Our results emphasize the importance of adaptive coordination in managing NREs effectively.

APPLICATION

This study provides valuable information for developing novel team training programs in health care that focus on adaptation to changing task requirements, for example, when faced with NREs.

Patient simulation: a literary synthesis of assessment tools in anesthesiology

High-fidelity patient simulation (HFPS) has been hypothesized as a modality for assessing competency of knowledge and skill in patient simulation, but uniform methods for HFPS performance assessment (PA) have not yet been completely achieved. Anesthesiology as a field founded the HFPS discipline and also leads in its PA. This project reviews the types, quality, and designated purpose of HFPS PA tools in anesthesiology. We used the systematic review method and systematically reviewed anesthesiology literature referenced in PubMed to assess the quality and reliability of available PA tools in HFPS. Of 412 articles identified, 50 met our inclusion criteria. Seventy seven percent of studies have been published since 2000; more recent studies demonstrated higher quality. Investigators reported a variety of test construction and validation methods. The most commonly reported test construction methods included "modified Delphi Techniques" for item selection, reliability measurement using inter-rater agreement, and intra-class correlations between test items or subtests. Modern test theory, in particular generalizability theory, was used in nine (18%) of studies. Test score validity has been addressed in multiple investigations and shown a significant improvement in reporting accuracy. However the assessment of predicative has been low across the majority of studies. Usability and practicality of testing occasions and tools was only anecdotally reported. To more completely comply with the gold standards for PA design, both shared experience of experts and recognition of test construction standards, including reliability and validity measurements, instrument piloting, rater training, and explicit identification of the purpose and proposed use of the assessment tool, are required.