Objective Structured Clinical Examination–Based Assessment of Regional Anesthesia Skills

Anesthesia assessment in the era of competences: state of the art

Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature  shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness. Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia. Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical  framework. Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.

Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes

Background

Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.

Methods

Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.

Results

Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.

Conclusions

Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments

Background

Simulation is an effective method for creating objective summative assessments of resident trainees. Real-time assessment (RTA) in simulated patient care environments is logistically challenging, especially when evaluating a large group of residents in multiple simulation scenarios. To date, there is very little data comparing RTA with delayed (hours, days, or weeks later) video-based assessment (DA) for simulation-based assessments of Accreditation Council for Graduate Medical Education (ACGME) sub-competency milestones. We hypothesized that sub-competency milestone evaluation scores obtained from DA, via audio-video recordings, are equivalent to the scores obtained from RTA.

Methods

Forty-one anesthesiology residents were evaluated in three separate simulated scenarios, representing different ACGME sub-competency milestones. All scenarios had one faculty member perform RTA and two additional faculty members perform DA. Subsequently, the scores generated by RTA were compared with the average scores generated by DA. Variance component analysis was conducted to assess the amount of variation in scores attributable to residents and raters.

Results

Paired t-tests showed no significant difference in scores between RTA and averaged DA for all cases. Cases 1, 2, and 3 showed an intraclass correlation coefficient (ICC) of 0.67, 0.85, and 0.50 for agreement between RTA scores and averaged DA scores, respectively. Analysis of variance of the scores assigned by the three raters showed a small proportion of variance attributable to raters (4% to 15%).

Conclusions

The results demonstrate that video-based delayed assessment is as reliable as real-time assessment, as both assessment methods yielded comparable scores. Based on a department’s needs or logistical constraints, our findings support the use of either real-time or delayed video evaluation for assessing milestones in a simulated patient care environment.

Simulation in pediatrics: Is it about time?

Pediatrics is a challenging field where “Time is Essence” and the interplay of time-bound dynamics has a huge influence on the outcomes, particularly in an acutely ill child. In this context, simulation based training appears to play a major role in training young Paediatricians to develop critical decision making skills and learning in a risk-free environment. In present times and in future, it is expected that simulation is used by practically every healthcare provider at some or multiple points in the training and certification process.

The opinion of post graduate students on objective structured clinical examination in Anaesthesiology: A preliminary report

Background and Aims:

The scenario in medical education is changing with objective structured clinical examination (OSCE) being introduced as an assessment tool. Its successful implementation in anaesthesiology postgraduate evaluation process is still limited. We decided to to evaluate the effectiveness of OSCE and compare it to conventional examinations as formative assessment tools in anaesthesiology.

Methods:

We conducted a cross-sectional comparative study in defined population of anaesthesiology postgraduate students to evaluate the effectiveness of OSCE as compared to conventional examination as formative assessment tool in anaesthesiology. Thirty-five students appeared for the conventional examination on the 1^st day and viva voce on the 2^nd day and OSCE on the last day. At the conclusion of the assessment, all the students were asked to respond to the perception evaluation questionnaire. We analysed the perception of OSCE among the students.

Results:

Results showed a positive perception of the objective structured physical examination (OSCE) as well as structured 9 (25.7%), fair 19 (54.2%) and unbiased 13 (37.1%) with more standardised scoring 9 (25.7%). The students perceived OSCE to be less stressful than other examination. Thirty-one (88.5%) students agreed that OSCE is easier to pass than conventional method and 29 (82.5%) commented that the degree of emotional stress is less in OSCE than traditional methods.

Conclusion:

OSCE is better evaluation tool when compared to conventional examination.

Simulation-based assessment to identify critical gaps in safe anesthesia resident performance

BACKGROUND

Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty.

METHODS

An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys.

RESULTS

Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice.

CONCLUSION

The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.

Designing and Implementing the Objective Structured Clinical Examination in Anesthesiology

Since its description in 1974, the Objective Structured Clinical Examination (OSCE) has gained popularity as an objective assessment tool of medical students, residents, and trainees. With the development of the anesthesiology residents’ milestones and the preparation for the Next Accreditation System, there is an increased interest in OSCE as an evaluation tool of the six core competencies and the corresponding milestones proposed by the Accreditation Council for Graduate Medical Education.

In this article the authors review the history of OSCE and its current application in medical education and in different medical and surgical specialties. They also review the use of OSCE by anesthesiology programs and certification boards in the United States and internationally. In addition, they discuss the psychometrics of test design and implementation with emphasis on reliability and validity measures as they relate to OSCE.

Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82

Over the past two decades, there has been an exponential and enthusiastic adoption of simulation in healthcare education internationally. Medicine has learned much from professions that have established programs in simulation for training, such as aviation, the military and space exploration. Increased demands on training hours, limited patient encounters, and a focus on patient safety have led to a new paradigm of education in healthcare that increasingly involves technology and innovative ways to provide a standardized curriculum. A robust body of literature is growing, seeking to answer the question of how best to use simulation in healthcare education. Building on the groundwork of the Best Evidence in Medical Education (BEME) Guide on the features of simulators that lead to effective learning, this current Guide provides practical guidance to aid educators in effectively using simulation for training. It is a selective review to describe best practices and illustrative case studies. This Guide is the second part of a two-part AMEE Guide on simulation in healthcare education. The first Guide focuses on building a simulation program, and discusses more operational topics such as types of simulators, simulation center structure and set-up, fidelity management, and scenario engineering, as well as faculty preparation. This Guide will focus on the educational principles that lead to effective learning, and include topics such as feedback and debriefing, deliberate practice, and curriculum integration - all central to simulation efficacy. The important subjects of mastery learning, range of difficulty, capturing clinical variation, and individualized learning are also examined. Finally, we discuss approaches to team training and suggest future directions. Each section follows a framework of background and definition, its importance to effective use of simulation, practical points with examples, and challenges generally encountered. Simulation-based healthcare education has great potential for use throughout the healthcare education continuum, from undergraduate to continuing education. It can also be used to train a variety of healthcare providers in different disciplines from novices to experts. This Guide aims to equip healthcare educators with the tools to use this learning modality to its full capability.

Training in trauma management: the role of simulation-based medical education

Simulation-based medical education (SBME) offers a safe and "mistake-forgiving" environment to teach and train medical professionals. The diverse range of medical-simulation modalities enables trainees to acquire and practice an array of tasks and skills. SBME offers the field of trauma training multiple opportunities to enhance the effectiveness of the education provided in this challenging domain. Further research is needed to better learn the role of simulation-based learning in trauma management and education.

The use of simulation in healthcare: from systems issues, to team building, to task training, to education and high stakes examinations

PURPOSE OF REVIEW

Simulation in healthcare is becoming increasingly used. This review will spotlight some of the uses of simulation in healthcare training.

RECENT FINDINGS

Previously, evaluation of simulation training was typically from evaluations from trainees. Recent articles, however, have linked simulation training to actual patient outcomes and demonstrated skill retention up to 1 year. Objective measurements have demonstrated positive effects on healthcare education, have been successfully used in high stakes examinations, and have uncovered systems and patient safety issues.

SUMMARY

This article will review some recent studies showing how simulation can have a positive effect on patient outcomes and skill retention, uncover systems issues related to patient safety, and how simulation can be used in credentialing, and other high stakes examinations.