An Italian version of the Ottawa Crisis Resource Management Global Rating Scale: a reliable and valid tool for assessment of simulation performance

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Preparing medical first responders for crises: a systematic literature review of disaster training programs and their effectiveness

BACKGROUND

Adequate training and preparation of medical first responders (MFRs) are essential for an optimal performance in highly demanding situations like disasters (e.g., mass accidents, natural catastrophes). The training needs to be as effective as possible, because precise and effective behavior of MFRs under stress is central for ensuring patients' survival and recovery. This systematic review offers an overview of scientifically evaluated training methods used to prepare MFRs for disasters. It identifies different effectiveness indicators and provides an additional analysis of how and to what extent the innovative training technologies virtual (VR) and mixed reality (MR) are included in disaster training research.

METHODS

The systematic review was conducted according to the PRISMA guidelines and focused specifically on (quasi-)experimental studies published between January 2010 and September 2021. The literature search was conducted via Web of Science and PubMed and led to the inclusion of 55 articles.

RESULTS

The search identified several types of training, including traditional (e.g., lectures, real-life scenario training) and technology-based training (e.g., computer-based learning, educational videos). Most trainings consisted of more than one method. The effectiveness of the trainings was mainly assessed through pre-post comparisons of knowledge tests or self-reported measures although some studies also used behavioral performance measures (e.g., triage accuracy). While all methods demonstrated effectiveness, the literature indicates that technology-based methods often lead to similar or greater training outcomes than traditional trainings. Currently, few studies systematically evaluated immersive VR and MR training.

CONCLUSION

To determine the success of a training, proper and scientifically sound evaluation is necessary. Of the effectiveness indicators found, performance assessments in simulated scenarios are closest to the target behavior during real disasters. For valid yet inexpensive evaluations, objectively assessible performance measures, such as accuracy, time, and order of actions could be used. However, performance assessments have not been applied often. Furthermore, we found that technology-based training methods represent a promising approach to train many MFRs repeatedly and efficiently. These technologies offer great potential to supplement or partially replace traditional training. Further research is needed on those methods that have been underrepresented, especially serious gaming, immersive VR, and MR.

Instruments to evaluate non-technical skills during high fidelity simulation: A systematic review

Introduction

High Fidelity Simulations (HFS) are increasingly used to develop Non-Technical Skills (NTS) in healthcare providers, medical and nursing students. Instruments to measure NTS are needed to evaluate the healthcare providers' (HCPs) performance during HFS. The aim of this systematic review is to describe the domains, items, characteristics and psychometric properties of instruments devised to evaluate the NTS of HCPs during HFS.

Methods

A systematic review of the literature was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Studies were retrieved from PubMed, Cinahl, Web of Science, Cochrane Library, ProQuest and PubPsych. Studies evaluating the measurement properties of instruments used to assess NTS during HFS training were included. Pairs of independent reviewers determined the eligibility, extracted and evaluated the data. Risk of bias and appraisal of the methodological quality of the studies was assessed using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist, and the quality of the evidence with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

Results

A total of 3,953 articles were screened. A total of 110 reports were assessed for eligibility and 26 studies were included. Studies were conducted in Europe/United Kingdom (n = 13; 50%), North America/Australia (n = 12; 46%) and Thailand (n = 1; 4%). The NTS instruments reported in this review included from 1 to 14 domains (median of 4, Q1 = 3.75, Q3 = 5) and from 3 to 63 items (median of 15, Q1 = 10, Q3 = 19.75). Out of 19 NTS assessment instruments for HFS, the Team Emergency Assessment Measure (TEAM) can be recommended for use to assess NTS. All the other instruments require further research to assess their quality in order to be recommended for use during HFS training. Eight NTS instruments had a positive overall rating of their content validity with at least a moderate quality of evidence.

Conclusion

Among a large variety of published instruments, TEAM can be recommended for use to assess NTS during HFS. Evidence is still limited on essential aspects of validity and reliability of all the other NTS instruments included in this review. Further research is warranted to establish their performance in order to be reliably used for HFS.

Validation in Spanish of the Ottawa scale for non-technical skills in health personnel in crisis situations

INTRODUCTION

Non-technical skills are related to morbi-mortality in medicine; it has been proposed that 46% of fatal outcomes are explained to limitations in non-technical skills and only 5% to technical skills deficiencies, however, there is no validated instrument or scale in spanish that allows its evaluation in the management of medical crisis.

OBJECTIVE

To evaluate the psychometric properties of a Spanish-adapted version of the "Ottawa crisis resource management (CRM) global rating scale (GRS)" in medical staff involved in critical decision-making based in high-fidelity simulation, which could be beneficial to impact patient safety and improve clinical outcomes.

METHODS

Transversal cultural instrument validation and adaptation study, included 91 participants who simulated a medical crisis between 2018 and 2019, and to whom the Spanish version of the CRM-GRS was applied in order to evaluate its psychometric properties.

RESULTS

A cultural adaptation with translation into Spanish of the CRM-GRS was made. Subsequently, the scale was applied to 91 participants. An internal consistency (Cronbach's alpha) greater than 0.9 was found in each dimension. The level of inter-rater reliability, evaluated by the interclass coefficient was 0.59 to 0.69, and test-retest reliability with an interclass coefficient greater than 0.7. The validity of the convergent construct was moderate (interclass coefficient between 0.6 and 0.7 for all domains) and the validity of the divergent construct between 0.4 and 0.5 was found adequate.

CONCLUSION

The translated and adapted Spanish version of the CRM-GRS in crisis had adequate internal consistency, reliability, and construct validity.

Validation in spanish of the Ottawa scale for non-techical skills in health personnel in crisis situations

INTRODUCTION

Non-technical skills are related to morbi-mortality in medicine; it has been proposed that 46% of fatal outcomes are explained to limitations in non-technical skills and only 5% to technical skills deficiencies, however, there is no validated instrument or scale in spanish that allows its evaluation in the management of medical crisis.

OBJECTIVE

To evaluate the psychometric properties of a Spanish-adapted version of the "Ottawa crisis resource management (CRM) global rating scale (GRS)" in medical staff involved in critical decision-making based in high-fidelity simulation, which could be beneficial to impact patient safety and improve clinical outcomes.

METHODS

Transversal cultural instrument validation and adaptation study, included 91 participants who simulated a medical crisis between 2018 and 2019, and to whom the Spanish version of the CRM-GRS was applied in order to evaluate its psychometric properties.

RESULTS

A cultural adaptation with translation into Spanish of the CRM-GRS was made. Subsequently, the scale was applied to 91 participants. An internal consistency (Cronbach's alpha) greater than 0.9 was found in each dimension. The level of inter-rater reliability, evaluated by the interclass coefficient was 0.59 to 0.69, and test-retest reliability with an interclass coefficient greater than 0.7. The validity of the convergent construct was moderate (interclass coefficient between 0.6 and 0.7 for all domains) and the validity of the divergent construct between 0.4 and 0.5 was found adequate.

CONCLUSION

The translated and adapted Spanish version of the CRM-GRS in crisis had adequate internal consistency, reliability, and construct validity.

Interprofessional collaboration between different health care professions in Emilia Romagna

BACKGROUND

Interprofessional collaboration in the healthcare sector contributes to the delivery of high quality and safe services to patients across different subdivisions of the healthcare system which is faced with constant challenges. The international literature offers a plethora of tools for assessing the collaboration between health workers, but only a few of these have been validated in the Italian language. One that has undergone such validation is the interprofessional collaboration (IPC) scale, which measures the perception of collaboration among health professionals. An advantage of this scale is that is addresses all workers within the system, and is not limited to specific professions. The aim of the present study was to apply the validated Italian version of the IPC scale, to a context different to the one used for its validation, to measure the level of collaboration between different health care workers.

METHOD

A questionnaire-based study was conducted on a sample consisting of 329 health professionals working at Azienda USL-IRCCS in Reggio Emilia. The categorical and continuous variables were analysed using descriptive statistics (frequencies, percentages and SD).

RESULTS

The IPC scale showed physicians to express the highest level of collaboration with other professionals, in line with the results of other studies in the literature. The values calculated for the factors "accommodation" and "communication" were higher than for "isolation", depicting a good level collaboration. The only case in which the isolation factor, which describes an absence of collaboration, was equal to the other two factors was in relation to the evaluation of midwives by nursing aides/orderlies.

CONCLUSIONS

In conclusion, the Italian version of the IPC scale provides a useful instrument for measuring interprofessional collaboration between workers in the healthcare sector. In the present study, it revealed a satisfactory level of collaboration between health professionals in an organization located in Emilia Romagna, Italy.

Cross-cultural adaptation and validation of two crisis resource management scales

BACKGROUND

Simulation training programs in crisis resource management must be evaluated using valid and reliable instruments. We translated into Spanish and linguistically validate The Mayo High Performance Teamwork Scale (MHPTS) and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS) non-technical skills assessment instruments.

METHOD

We performed a standardised cross-cultural adaptation process. The psychometric properties of both instruments in their versions adapted to Spanish were subsequently evaluated in a sample of 100 students by using exploratory factor analysis and assessing internal consistency and convergent validity through a total of 94 simulation scenarios in urgent medical situations RESULTS: Our results for the MHPTS showed a one-dimensional structure containing 8 items which explained a total variance of 72.84%; the Ottawa GRS also had a one-dimensional structure, this time with 5 items, which explained a total variance of 91.79%. According to the Cronbach alpha, the internal consistency for the MHPTS was 0.94 (1-8 items) and 0.98 for the Ottawa GRS. In addition, there was a strong correlation between the MHPTS and Ottawa GRS (r = 0.97; p < 0.001).

CONCLUSION

We found strong evidence for the high validity and reliability of the Spanish versions of both these tools when tested in Spanish simulated emergency contexts.

Residents working with Médecins Sans Frontières: training and pilot evaluation

Background

Well-prepared humanitarian workers are now more necessary than ever. Essential to the preparation process are: clearly defined learning objectives, curricula tailored to the nuances of humanitarian settings, simulation-based training, and evaluation. This manuscript describes a training program designed to prepare medical residents for their first field deployment with Médecins Sans Frontières and presents the results of a pilot assessment of its effectiveness.

Methods

The training was jointly developed by the Research Center in Emergency and Disaster Medicine- CRIMEDIM of the Università del Piemonte Orientale, Novara, Italy, and the humanitarian aid organization Médecins Sans Frontières- Italy (MSF-Italy); the following topics were covered: disaster medicine, public health, safety and security, infectious diseases, psychological support, communication, humanitarian law, leadership, and job-specific skills. It used a blended-learning approach consisting of a 3-month distance learning module; 1-week instructor-led coaching; and a field placement with MSF. We assessed its effectiveness using the first three levels of Kirkpatrick’s training evaluation model.

Results

Eight residents took part in the evaluation. Four were residents in emergency medicine, 3 in anesthesia, and 1 in pediatrics; 3 of them were female and the median age was 31 years. Two residents were deployed in Pakistan, 1 in Afghanistan, 1 in the Democratic Republic of Congo, 1 in Iraq, 2 in Haiti and 1 on board of the MSF Mediterranean search & rescue ship. Mean deployment time was 3 months. The average median score for the overall course was 5 (excellent). There was a significant improvement in post-test multiple choice scores (p = 0.001) and in residents’ overall performance scores (P = 0.000001).

Conclusion

Residents were highly satisfied with the training program and their knowledge and skills improved as a result of participation.

Trial registration

This study was approved by the Institutional Ethics Committee (date 24-02-2016, study code UPO.2015.4.10).

Cognitive appraisals and team performance under stress: A simulation study

OBJECTIVES

The present study explored how challenge and threat responses to stress relate to performance, anxiety, confidence, team identity and team characteristics (time spent in training and postgraduate experience) in a medical simulation-based team competition.

METHODS

The study was conducted during a national simulation-based training event for residents, the SIMCUP Italia 2018. The SIMCUP is a simulation competition in which teams of four compete in simulated medical emergency scenarios. Cross-sectional data were collected prior to the 3 days of the competition. Subjects included 95 participants on 24 teams. Before the competition on each day, participants completed brief self-report measures that assessed demands and resources (which underpin challenge and threat responses to stress), cognitive and somatic anxiety, self-confidence and team identification. Participants also reported time (hours) spent practising as a team and years of postgraduate experience. A team of referees judged each scenario for performance and assigned a score. A linear mixed model using demands and resources was built to model performance.

RESULTS

The data showed that both demands and resources have positive effects on performance (31 [11-50.3] [P < .01] and 54 [25-83.3] [P < .01] percentage points increase for unitary increases in demands and resources, respectively); however, this is balanced by a negative interaction between the two (demands * resources interaction coefficient = -10 [-16 to -4.2]). A high level of resources is associated with better performance until demands become very high. Cognitive and somatic anxieties were found to be correlated with demands (Pearson's r = .51 [P < .01] and Pearson's r = .48 [P < .01], respectively). Time spent training was associated with greater perceptions of resources (Pearson's r = .36 [P < .01]).

CONCLUSIONS

We describe a model of challenge and threat that allows for the estimation of performance according to perceived demands and resources, and the interaction between the two. Higher levels of resources and lower demands were associated with better performance.

Comparing Resource Management Skills in a High- versus Low-Resource Simulation Scenario: A Pilot Study

BACKGROUND

Low-resource environments, such as those found in humanitarian crises, pose significant challenges to the provision of proper medical treatment. While the lack of training of health providers to such settings has been well-acknowledged in literature, there has yet to be any scientific evidence for this phenomenon.

METHODS

This pilot study utilized a randomized crossover experimental design to examine the effects of high- versus low-resource simulated scenarios of a resuscitation of a critically ill obstetric patient on a medical doctors' performance and inter-personal skills. Ten senior residents (fifth-year post-graduate) of the Maggiore Hospital School of Medicine (Novara, NO, Italy) were included in the study.

RESULTS

Overall performance score for the high-resource setting was 5.2, as opposed to only 2.3 for the low-resource setting. The mean effect size for the overall score was 2.9 (95% CI, 1.7-4.0; P <.001). The results suggest a significant decrease in both technical (medical) and non-technical skills, such as leadership, problem solving, situation awareness, resource utilization, and communication in the low-resource environment setting. The latter finding is of special important since it was yet to be reported.

CONCLUSIONS

This pilot study suggests that untrained physicians in low-resource environments may experience a considerable setback not only to their professional performance, but also to their interpersonal skills, when deployed ill-prepared to humanitarian missions. Consequently, this may endanger the health of local populations.

A novel simulation competition format as an effective instructional tool in post-graduate medical education

Objective

Medical simulation competitions are a growing reality. This study aims at exploring if a novel format of simulation competition (SIMCUP) can be an effective educational format in post-graduate education.

Design

We designed a 2-day event that included scientific educational lectures, an orientation to the competition, familiarization with the simulation lab, and competition time. Day 1 was devoted to preliminary rounds and was structured using an Objective Structured Clinical Examination (OSCE)-like system. On day 2, the first four teams advanced to semi-finals and then to finals, which were held using a classical SimWars style.

Setting and subjects

A total of 14 four-participant teams participated in the event over two editions (Ed.1 in 2015 and Ed.2 in 2016).

Interventions

External referees evaluated both technical and non-technical skills for each simulated scenario. Each participant was also administered pre- and post-test questionnaires covering self-perception about the confidence in managing simulated clinical cases, educational effectiveness, satisfaction with the simulation experience, and previous simulation training.

Main results

Overall participants found SIMCUP a useful learning experience, rating it 10 [9, 10] and 10 [7.75–10] out of 10 for Ed.1 and Ed.2, respectively. Participants reported, using a 10-point semantic differential scale ranging from “1 - strongly disagree.” to “10 - strongly agree,” finding both days to be educationally effective: day 1 was rated 9 [7–10] and 9 [8–10] as day 2 was rated 8 [7–10] and 8 [7–10] for Ed. 1 and Ed. 2, respectively.

Participants’ self-perception regarding the confidence of managing the specific scenarios significantly improved immediately after the event as measured by pre- and post-questionnaires for all stations and during both editions.

Conclusion

This study suggests that simulation competition can serve as an effective instructional format in residency training.

Electronic supplementary material

The online version of this article (10.1186/s41077-018-0075-4) contains supplementary material, which is available to authorized users.

Anesthesiology Resident Induction Month: a pilot study showing an effective and safe way to train novice residents through simulation

BACKGROUND

The transition of new residents from medical school to the post-graduate clinical environment remains challenging. We hypothesized that an introductory simulation course could improve new residents' performance in anesthesiology.

METHODS

The Anesthesiology Residents Induction Month (ARIM) program was designed as a non-clinical simulation training program aiming at providing the theoretical and practical skills to safely approach, as junior anesthesiologists, the operating rooms. For each participant, specific knowledge, procedural skills and non-technical performance were assessed with a pre and post-test approach, before and immediately after the participation in the study.

RESULTS

Fifteen first-month residents participated in the study. As compared to pre-test, residents significantly improved in all three evaluated areas. Pre-test knowledge assessment mean improved from 56% to 73% in the post-test (P<0.001). In the procedural skills assessment, pre-test mean improved from 43% to 77% (P<0.001) and non-technical skills assessment improved from 3.17 to 4.61 (in a scale out of seven points) in the post-test (P<0.001).

CONCLUSIONS

Data suggest that an intensive simulation-based program can be an effective way for first-year residents to rapidly acquire and develop basic skills specific to anesthesiology. There might be benefits to begin residency with a training program aiming at developing and standardizing technical and non-technical skills.