Development and applications of the Anaesthetists' Non-Technical Skills behavioural marker system: a systematic review

OBJECTIVES

To comprehensively synthesise evidence regarding the validity and reliability of the Anaesthetists' Non-Technical Skills (ANTS) behavioural marker system and its application as a tool for the training and assessment of non-technical skills to improve patient safety.

DESIGN

Systematic review.

DATA SOURCES

We employed a citation search strategy. The Scopus and Web of Science databases were searched for articles published from 2002 to May 2022.

ELIGIBILITY CRITERIA

English-language publications that applied the ANTS system in a meaningful way, including its use to guide data collection, analysis and reporting.

DATA EXTRACTION AND SYNTHESIS

Study screening, data extraction and quality assessment were performed by two independent reviewers. We appraised the quality of included studies using the Joanna Briggs Institute Critical Appraisal Checklists. A framework analysis approach was used to summarise and synthesise the included articles.

RESULTS

54 studies were identified. The ANTS system was applied across a wide variety of study objectives, settings and units of analysis. The methods used in these studies varied and included quantitative (n=42), mixed (n=8) and qualitative (n=4) approaches. Most studies (n=47) used the ANTS system to guide data collection. The most commonly reported reliability statistic was inter-rater reliability (n=35). Validity evidence was reported in 51 (94%) studies. The qualitative application outcomes of the ANTS system provided a reference for the analysis and generation of new theories across disciplines.

CONCLUSION

Our results suggest that the ANTS system has been used in a wide range of studies. It is an effective tool for assessing non-technical skills. Investigating the methods by which the ANTS system can be evaluated and implemented for training within clinical environments is anticipated to significantly enhance ongoing enhancements in staff performance and patient safety.

PROSPERO REGISTRATION NUMBER

CRD42022297773.

Effect evaluation of competency-based education (CBE) combined with multi-disciplinary team (MDT) teaching mode in respiratory rehabilitation nursing teaching：A randomized controlled trial

AIM

To explore the application effect of competency-based education (CBE) combined with multi-disciplinary team (MDT) teaching mode in respiratory rehabilitation nursing teaching.

BACKGROUND

Respiratory rehabilitation, as an important non drug treatment for chronic respiratory disease, started late in China, with low implementation rate in hospitals and insufficient knowledge of nursing staff. Therefore, it is urgent to cultivate high professional level nurses with strong professional ability.

DESIGN

A randomized controlled trial.

METHODS

We selected nurses from the respiratory and critical care department at a Grade III, Level A hospital in Beijing, who participated in the respiratory rehabilitation training program between March 2020 and August 2022, as our study participants. Participants were grouped using a random number method. Nurses who participated in the study from March 2020 to May 2021 were set as the control group, and the nurses who participated in the study from June 2021 to August 2022 were set as the test group. The control group used traditional teaching methods, and the test group used the CBE combined MDT teaching model. At the end of the test, the theory, operating skills, satisfaction and core competence were assessed.

RESULTS

After training, the theoretical examination score of the test group (93.71 ± 1.94) was higher than that of the control group (92.37 ± 1.92), the operational examination score of the test group (93.11 ± 2.12) was higher than that of the control group (91.61 ± 1.93), the overall teaching satisfaction of the test group (4.45 ± 0.50) was higher than that of the control group (4.13 ± 0.57), and the total score of the core competence of the test group (148.73 ± 7.08) was higher than that of the control group (141.02 ± 6.41). The difference between the two groups was statistically significant (P<0.05).

CONCLUSION

CBE combined with MDT teaching mode has a good effect in Pulmonary and Critical Care Medicine (PCCM) respiratory rehabilitation nursing teaching.

Continuing professional development (CPD) for anesthetists: A systematic review

BACKGROUND

In accordance with the focus on patient safety and quality in healthcare, continuing professional development (CPD) has received increasing levels of attention as a means to ensure physicians maintain their clinical competencies and are fit to practice. There is some evidence of a beneficial effect of CPD, though few studies have evaluated its effect within anesthesia. The primary aim of this systematic review was to establish which CPD activities anesthetists are engaged in and their effectiveness. The secondary aim was to explore which methods are employed to evaluate anesthetists' clinical performance.

METHODS

Databases searched: Medline, Embase and Web of Science, in May 2023. Additional papers were identified through searching the references of included studies. Eligible studies included anesthetists, either exclusively or combined with other healthcare professionals, who underwent a learning activity or assessment method as part of a formalized CPD program or a stand-alone activity. Non-English language studies, non-peer reviewed studies and studies published prior to 2000 were excluded. Eligible studies were quality assessed and narratively synthesized, with results presented as descriptive summaries.

RESULTS

A total of 2112 studies were identified, of which 63 were eligible for inclusion, encompassing more than 137,518 participants. Studies were primarily of quantitative design and medium quality. Forty-one studies reported outcomes of single learning activities, whilst 12 studies investigated different roles of assessment methods in CPD and ten studies evaluated CPD programs or combined CPD activities. A 36 of the 41 studies reported positive effects of single learning activities. Investigations of assessment methods revealed evidence of inadequate performance amongst anesthetists and a mixed effect of feedback. Positive attitudes and high levels of engagement were identified for CPD programs, with some evidence of a positive impact on patient/organizational outcomes.

DISCUSSION

Anesthetists are engaged in a variety of CPD activities, with evidence of high levels of satisfaction and a positive learning effect. However, the impact on clinical practice and patient outcomes remains unclear and the role of assessment is less well-defined. There is a need for further, high-quality studies, evaluating a broader range of outcomes, in order to identify which methods are most effective to train and assess specialists in anesthesia.

Collecting evidence of validity for an assessment tool for Norwegian medical students' non-technical skills (NorMS-NTS): usability and reliability when used by novice raters

BACKGROUND

The NorMS-NTS tool is an assessment tool for assessing Norwegian medical students' non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters.

METHODS

This study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools' internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool's usability.

RESULTS

The ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77-0.91. Cronbach's alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching.

CONCLUSIONS

The observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students' non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

Cross-cultural adaptation and validation of the Spanish version of the Anesthetists' Non-Technical Skills (ANTS) assessment tool

STUDY OBJECTIVE

Establish the transcultural validity of Anesthetists Non-Technical Skills (ANTS) in a Spanish-speaking country.

DESIGN

Prospective cohort.

SETTING

Clinical simulation center.

SUBJECTS

Forty-two Anesthesia PY2 and PY3 residents participated in the study.

INTERVENTIONS

Four clinical scenarios simulating anesthesia crises were assessed with a Spanish version of ANTS. Every simulated scenario was run twice with a time span of 3 to 4 months between them.

MEASUREMENTS

Two anesthesiologists independently assessed all simulated sessions using ANTS. The ANTS indicators of construct validity were obtained by confirmatory factor analysis. Various goodness-of-fit indices of the factorial model were calculated: Comparative Fit Index (CFI); Tucker-Lewis Adjustment Index (TLI) and Root Mean Square Error of Approximation (RMSEA). The standardized factor loadings and the determination coefficient (R2) was also estimated.

MAIN RESULTS

A total of 212 clinical scenarios were analyzed. The specified factorial model had the same grouping of elements in four domains as the original version of ANTS. The CFI index and the TLI were 0.99 and the RMSEA reached 0.07 (95% CI 0.06-0.08). All the standardized factor loadings were found to be >0.4. Also, the elements obtained an R2 value that fluctuated between 0.54 and 0.92.

CONCLUSIONS

The Spanish version of ANTS is a valid, reliable and a useful tool to assess non-technical skills in Spanish-speaking countries. The applicability of the instrument was comparable to the original setting. The high reliability of ANTS in our setting allows us to propose its use not just in an educational and research setting; it can be used as an assessment tool of non-technical skills.

Competency-Based Medical Education: Are Canadian Pediatric Anesthesiologists Ready?

Background

With the introduction of Competency-Based Medical Education (CBME), the Canadian Pediatric Anesthesia Society (CPAS) surveyed its members to assess their awareness of and prior experience with CBME concepts and evaluation tools, and identify methods for faculty development of CBME teaching strategies for pediatric anesthesia residents and fellows.

Methods

An online survey was sent to CPAS members. Outcomes included respondents’ previous exposure to CBME and the educational support they had received in anticipation of the curriculum. Questions used multi-item Likert scales and a general feedback question. 

Results

The response rate was 39% (60/155). Eighty-eight percent of respondents spent ≥50% of their time practicing pediatric anesthesia; 78% and 45% spent at least a quarter of their time teaching residents and fellows respectively. Eighty-three percent were familiar with CBME concepts, and 58% were familiar with Milestones, Competencies, and Entrustable Professional Activities (EPAs). However, 64% had not received any formal training and 52% had not used any CBME evaluation tools. Learning preferences included small group discussions (72%), lectures with questions and answers (Q&A) (62%), seminars (50%), and workshops (50%).

Conclusions

Despite widespread awareness of CBME concepts, there is a need to educate Canadian pediatric anesthesiologists regarding CBME evaluation tools. Faculty development support will increase the utilization of these tools in teaching practice.

Teaching non-technical skills: the patient centered approach

The surgical setting is a highly complex environment where, in ideal conditions, everything should be under control to ensure a positive outcome. However, the existing complexity opens the possibility for multiple failures along the process and many of those failures are related to what is call the non-technical skills of the members of the team. We cannot eradicate human error, but we can try to avoid future mistakes in our daily practice introducing the awareness for providing a high-quality care in which patient safety is crucial. This paper presents an easy approach to concepts and teaching possibilities of those non-technical skills.

Development and Evaluation of the Taxonomy of Trauma Leadership Skills-Shortened for Observation and Reflection in Training: A Practical Tool for Observing and Reflecting on Trauma Leadership Performance

INTRODUCTION

Trauma leadership skills are increasingly being addressed in trauma courses, but few resources are available to systematically observe and debrief trainees' performances. The authors therefore translated their previously developed, extensive Taxonomy of Trauma Leadership Skills (TTLS) into a practical observation tool that is tailored to the vocabulary of clinician instructors and their workflow and workload during simulation-based training.

METHODS

In 2016 to 2018, the TTLS was subjected to practical evaluation in an iterative process of 2 stages. In the first stage, testing panels of trauma specialists observed excerpts from videotaped simulations and indicated from the list of elements which behaviors they felt were being shown. Any ambiguities or redundancy were addressed by rephrasing or combining elements. In the second stage, iterations were used in actual scenario training to observe and debrief trainees' performances. The instructors' recommendations resulted in further improvements of clarity, ease of use, and usefulness, until no new suggestions were raised.

RESULTS

The resultant "TTLS-Shortened for Observation and Reflection in Training" was given a simpler structure and more concrete and self-explanatory benchmarks. It contains 6 skill categories for evaluation, each with 4 to 6 benchmark behaviors.

CONCLUSIONS

The TTLS-Shortened for Observation and Reflection in Training is an important addition to other trauma assessment tools because of its specific focus on leadership skills. It helps set concrete performance expectations, simplify note taking, and target observations and debriefings. One central challenge was striking a balance between its conciseness and specificity. The authors reflected on how the decisions for the resultant structure ease and leverage the conduct of observations and performance debriefing.

Psychometric Testing of a Structured Assessment Instrument for Non-technical Skills (NANTS-no) for Use in Clinical Supervision of Student Nurse Anesthetists

BACKGROUND

This study evaluated psychometric properties of a structured behavioral assessment instrument, Nurse Anaesthetists' Non-Technical Skills-Norway (NANTS-no). It estimated whether reliable assessments of nontechnical skills (NTS) could be made after taking part in a workshop. An additional objective was to evaluate the instrument's acceptability and usability.

METHODS

An explorative design was used. Nurse anesthetists (n = 46) involved in clinical supervision attended a 6-hour workshop on NTS, then rated NTS in video-recorded simulated scenarios and completed a questionnaire.

RESULTS

High reliability and dependability were estimated in this setting. Participants regarded the instrument as useful for clinical supervision of student nurse anesthetists (SNAs).

CONCLUSIONS

Findings suggest that NANTS-no may be reliable for performing clinical assessments of SNAs and encouraging critical reflection. However, further research is needed to explore its use in clinical settings.

Reliability of assessment of medical students’ non-technical skills using a behavioural marker system: does clinical experience matter?

Introduction

Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students’ Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students’ non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience.

Methods

Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters’ levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC).

Results

Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65.

Conclusions

Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.

Anesthesia assessment in the era of competences: state of the art

Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature  shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness. Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia. Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical  framework. Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.

An observational study of technical and non-technical skills in advanced life support in the clinical setting

OBJECTIVE

Technical skills (TS) and non-technical skills (NTS) are the primary elements ensuring patient safety during advanced life support (ALS) and effective crisis resource management (CRM). Both skills are needed to perform high-quality ALS, though they are traditionally practiced separately. The evidence of the association between NTS and TS in high-quality ALS performance is insufficient. Hence, we aimed to evaluate the association between the skills in real-life in-hospital ALS situations.

METHODS

We video recorded real-life in-hospital ALS situations, analyzed TS and NTS demonstrated in them with an instrument measuring TS and NTS, and tested the linear association between NTS and TS using a linear mixed model.

RESULTS

Among 50 real-life in-hospital ALS situations that we recorded, 20 had adequate data for analysis. NTS and TS total scores were associated with one another (slope 0.48, P < 0.001). All NTS subcategories were associated with the TS total score (slopes ranging from 0.29 to 0.39, P < 0.001). The NTS total score and TS subcategories (chest compression quality, ventilation quality, rhythm control and defibrillation quality) were associated with one another (slopes ranging from 0.37 to 0.56, P < 0.01).

CONCLUSIONS

The resuscitation teams who demonstrated good NTS also performed the technical aspects of ALS better. The results suggest that NTS and TS have an association with one another in real-life in-hospital ALS situations. NTS performance had the most evident association with chest compression quality and rhythm control and defibrillation quality; these are considered the most crucial elements affecting outcomes of ALS. The findings of the study present novel information of what and why to emphasize in ALS training.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov, NCT03017144.

The Extended Supervised Learning Event (ESLE): Assessing Nontechnical Skills in Emergency Medicine Trainees in the Workplace

STUDY OBJECTIVE

The contribution of emergency medicine clinicians' nontechnical skills in providing safe, high-quality care in the emergency department (ED) is well known. In 2015, the UK Royal College of Emergency Medicine introduced explicit validated descriptors of nontechnical skills needed to function effectively in the ED. A new nontechnical skills assessment tool that provided a score for 12 domains of nontechnical skills and detailed narrative feedback, the Extended Supervised Learning Event (ESLE), was introduced and was mandated as part of the Royal College of Emergency Medicine assessment schedule. We aim to evaluate the psychometric reliability of the ESLE in its first year of use.

METHODS

ESLEs were mandated for all UK emergency medicine trainees in the final 4 years of a 6-year national training program from August 2015. The completed assessments were uploaded to the Royal College of Emergency Medicine e-portfolio. All assessments recorded in the Royal College of Emergency Medicine e-portfolio database between August 2015 and August 2016 were anonymized and analyzed for psychometric reliability, using generalizability theory. Decision analysis was used to model the effect of altering the number of episodes and assessors on reliability.

RESULTS

A total of 1,390 ESLEs were analyzed. The majority (62%) of the variation in nontechnical skills scores was attributable to the trainee's ability. The circumstances of the event (eg, case complexity, workload) accounted for 21% and the stringency or leniency of assessors the remaining 16%. Decision analysis suggests that 3 ESLEs by 2 or more assessors, as currently recommended in the Royal College of Emergency Medicine curriculum, provide an assessment with a reliability coefficient of 0.8.

CONCLUSION

Board-certified-equivalent emergency medicine supervisors are able to provide reliable assessments of emergency medicine trainees' nontechnical skills in the workplace by using the ESLE.

Impact of sleep deprivation on anaesthesia residents' non-technical skills: a pilot simulation-based prospective randomized trial

Background

Sleep deprivation is common in anaesthesia residents, but its impact on performance remains uncertain. Non-technical skills (team working, situation awareness, decision making, and task management) are key components of quality of care in anaesthesia, particularly in crisis situations occurring in the operating room. The impact of sleep deprivation on non-technical skills is unknown. We tested the hypothesis that in anaesthesia residents sleep deprivation is associated with impaired non-technical skills.

Methods

Twenty anaesthesia residents were randomly allocated to undergo a simulation session after a night shift [sleep-deprived (SLD) group, n =10] or after a night of rest [rested (R) group, n =10] from January to March 2015. The simulated scenario was a situation of crisis management in the operating room. The primary end point was a composite score of anaesthetists' non-technical skills (ANTS) assessed by two blinded evaluators.

Results

Non-technical skills were significantly impaired in the SLD group [ANTS score 12.2 (interquartile range 10.5-13)] compared with the R group [14.5 (14-15), P <0.02]. This difference was mainly accounted for by a difference in the team working item. On the day of simulation, the SLD group showed increased sleepiness and decreased confidence in anaesthesia skills.

Conclusions

In this randomized pilot trial, sleep deprivation was associated with impaired non-technical skills of anaesthesia residents in a simulated anaesthesia intraoperative crisis scenario.

Trial registration

NCT02622217.

Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

BACKGROUND

Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

METHODS

Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

RESULTS

The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

CONCLUSIONS

Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

Evaluation of a Simpler Tool to Assess Nontechnical Skills During Simulated Critical Events

INTRODUCTION

Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale.

METHODS

Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall.

RESULTS

The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74.

CONCLUSIONS

Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.

The reliability and usability of the Anesthesiologists' Non-Technical Skills (ANTS) system in simulation research

Background

Non-technical skills (NTS) such as leadership and team work are important in providing good quality of care. One system to assess physicians’ NTS is the Anesthesiologists’ Non-Technical Skills (ANTS) system. The present study evaluates the ANTS system on the interrater reliability and usability for research purposes.

Methods

Ten anesthesiologists and 20 anesthesiology residents performed two resuscitation scenarios (with and without the presence of distractors) in a simulation room with a full-scale patient simulator. The scenarios were videotaped. Two independent raters rated the NTS of the anesthesiologists using the ANTS system. The intraclass correlation coefficients (ICC) were calculated to determine the interrater reliability of both the total NTS score and the measured differences between the two scenarios. The raters filled out a questionnaire to obtain insights in the usability of the ANTS system for research purposes.

Results

The ICC for the total score of the NTS was substantial (0.683), and the ICC of the elements varied between 0.371 for assessing capabilities and 0.670 for providing and maintaining standards. The intraclass correlation coefficient of measuring differences was fair (0.502). The raters judged the usability as good.

Conclusions

The ANTS system was reliable for the total score and usable to measure physicians’ NTS in a research setting. However, there was variation between the reliability of the elements. We recommend that if the ANTS is used for research, a pilot study should determine elements not applicable or observable in the scenario of interest; these elements should be excluded from the study.

Electronic supplementary material

The online version of this article (doi:10.1186/s41077-016-0013-2) contains supplementary material, which is available to authorized users.

Exploring the relationship between anaesthesiologists' non-technical and technical skills

BACKGROUND

A combination of non-technical skills (NTS) and technical skills (TS) is crucial for anaesthetic patient management. However, a deeper understanding of the relationship between these two skills remains to be explored. We investigated the characteristics of trainee anaesthesiologists' NTS and TS in a simulated unexpected difficult airway management scenario.

METHODS

A mixed-method approach was used to explore the relationship between NTS and TS in 25 videos of 2nd year trainee anaesthesiologists managing a simulated difficult airway scenario. The videos were assessed using the customised version of the Anaesthetists' Non-Technical Skills System, ANTSdk, and an adapted TS checklist for calculating the correlation between NTS and TS. Written descriptions of the observed NTS were analysed using directed content analysis.

RESULTS

The correlation between the NTS and the TS ratings was 0.106 (two-tailed significance of 0.613). Inter-rater reliability was substantial. Themes characterising good NTS included a systematic approach, planning and communicating decisions as well as responding to the evolving situation. A list of desirable, concrete NTS for the specific airway management situation was generated.

CONCLUSION

This study illustrates that anaesthesiologist trainees' NTS and TS were not correlated in this setting, but rather intertwined and how the interplay of NTS and TS can impact patient management. Themes describing the characteristics of NTS and a list of desirable, concrete NTS were developed to aid the understanding, training and use of NTS.

Lessons from surgery and anaesthesia: evaluation of non-technical skills in interventional radiology

In the medical profession, surgery and anaesthesia are leading the way in identifying human errors that negatively affect patient safety. Evidence suggests that the implementation of non-technical skills assessments reduces such errors. Interventional Radiology is a procedural based speciality and therefore may also benefit from formal assessment of non-technical skills. This literature review supports the use of standardised assessment tools used in surgery and anaesthesia. Using the Downing framework of internal validity, the tools demonstrated good internal consistency but a spectrum of inter-rater variability, which can be partially improved with training. At present, a formal Interventional Radiology non-technical skills assessment tool is probably not suitable to be a stand-alone ‘high stakes’ assessment, but may be a useful adjunct to the existing array of workplace-based assessments.

Non-technical skills of anaesthesia providers in Rwanda: an ethnography

INTRODUCTION

Patient safety depends on excellent practice of anaesthetists' non-technical skills (ANTS). The ANTS framework has been validated in developed countries but there is no literature on the practice of ANTS in low-income countries. This study examines ANTS in this unexplored context.

METHODS

This qualitative ethnographic study used observations of Rwandan anaesthesia providers and in-depth interviews with both North American and Rwandan anaesthesia providers to understand practice of ANTS in Rwanda.

RESULTS

Communication is central to the practice of ANTS. Cultural factors in Rwanda, such as lack of assertiveness and discomfort taking leadership, and the strains of working in a resource-limited environment hinder the unfettered and focused communication needed for excellent anaesthesia practice.

CONCLUSION

Despite the challenges, anaesthesia providers are able to coordinate activities when good communication is actively encouraged. Future teaching interventions should address leadership and communication skills through encouraging both role definition and speaking up for patient safety.

A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application?

OBJECTIVE

Behavioural marker systems are advocated as a method for providing accurate assessments, directing feedback and determining the impact of teamwork improvement initiatives. The present article reports on the state of quality surrounding their use in healthcare and discusses the implications of these findings for future research, development and application. In doing so, this article provides a practical resource where marker systems can be selected and evaluated based on their strengths and limitations.

METHODS

Four research questions framed this review: what are the attributes of behavioural marker systems? What evidence of reliability and validity exists? What skills and expertise are required for their use? How have they been applied to investigate the relationship between teamwork and other constructs?

RESULTS

Behavioural markers systems are generally designed for specific work domains or tasks. They often cover similar content with inconsistent terminology, which complicates the comparison of research findings across clinical domains. Although several approaches were used to establish the reliability and validity of marker systems, the marker system literature, as a whole, requires more robust reliability and validity evidence. The impact of rater training on rater proficiency was mixed, but evidence suggests that improvements can be made over time.

CONCLUSIONS

A consensus of definitions for teamwork constructs must be reached to ensure that the meaning behind behavioural measurement is understood across disciplines, work domains and task types. Future development efforts should focus on the cost effectiveness and feasibility of measurement tools including time spent training raters. Further, standards for the testing and reporting of psychometric evidence must be established. Last, a library of tools should be generated around whether the instrument measures general or domain-specific behaviours.

Teamwork assessment in internal medicine: a systematic review of validity evidence and outcomes

OBJECTIVE

Valid teamwork assessment is imperative to determine physician competency and optimize patient outcomes. We systematically reviewed published instruments assessing teamwork in undergraduate, graduate, and continuing medical education in general internal medicine and all medical subspecialties.

DATA SOURCES

We searched MEDLINE, MEDLINE In-process, CINAHL and PsycINFO from January 1979 through October 2012, references of included articles, and abstracts from four professional meetings. Two content experts were queried for additional studies.

STUDY ELIGIBILITY

Included studies described quantitative tools measuring teamwork among medical students, residents, fellows, and practicing physicians on single or multi-professional (interprofessional) teams.

STUDY APPRAISAL AND SYNTHESIS METHODS

Instrument validity and study quality were extracted using established frameworks with existing validity evidence. Two authors independently abstracted 30 % of articles and agreement was calculated.

RESULTS

Of 12,922 citations, 178 articles describing 73 unique teamwork assessment tools met inclusion criteria. Interrater agreement was intraclass correlation coefficient 0.73 (95 % CI 0.63-0.81). Studies involved practicing physicians (142, 80 %), residents/fellows (70, 39 %), and medical students (11, 6 %). The majority (152, 85 %) assessed interprofessional teams. Studies were conducted in inpatient (77, 43 %), outpatient (42, 24 %), simulation (37, 21 %), and classroom (13, 7 %) settings. Validity evidence for the 73 tools included content (54, 74 %), internal structure (51, 70 %), relationships to other variables (25, 34 %), and response process (12, 16 %). Attitudes and opinions were the most frequently assessed outcomes. Relationships between teamwork scores and patient outcomes were directly examined for 13 (18 %) of tools. Scores from the Safety Attitudes Questionnaire and Team Climate Inventory have substantial validity evidence and have been associated with improved patient outcomes.

LIMITATIONS

Review is limited to quantitative assessments of teamwork in internal medicine.

CONCLUSIONS

There is strong validity evidence for several published tools assessing teamwork in internal medicine. However, few teamwork assessments have been directly linked to patient outcomes.

Utility and assessment of non-technical skills for rapid response systems and medical emergency teams

Efforts are ongoing to improve outcomes from cardiac arrest and medical emergencies. A promising quality improvement modality is use of non-technical skills (NTS) that aim to address human factors through improvements in performance of leadership, communication, situational awareness and decision-making. Originating in the airline industry, NTS training has been successfully introduced into anaesthesia, surgery, emergency medicine and other acute medical specialities. Some aspects of NTS have already achieved acceptance for cardiac arrest teams. Leadership skills are emphasised in advanced life support training and have shown favourable results when employed in simulated and clinical resuscitation scenarios. The application of NTS in medical emergency teams as part of a rapid response system attending medical emergencies is less certain; however, observations of simulations have also shown promise. This review highlights the potential benefits of NTS competency for cardiac arrest teams and, more importantly, medical emergency teams because of the diversity of clinical scenarios encountered. Discussion covers methods to assess and refine NTS and NTS training to optimise performance in the clinical environment. Increasing attention should be applied to yielding meaningful patient and organisational outcomes from use of NTS. Similarly, implementation of any training course should receive appropriate scrutiny to refine team and institutional performance.

Assessing the patient safety competencies of healthcare professionals: a systematic review

Background

Patient safety training of healthcare professionals is a new area of education. Assessment of the pertinent competencies should be a part of this education. This review aims to identify the available assessment tools for different patient safety domains and evaluate them according to Miller's four competency levels.

Methods

The authors searched PubMed, MEDLINE, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science, psycINFO and the Education Resource Information Center (ERIC) from the start of each database to December 2010 for English-language articles that evaluated or described tools for the assessment of the safety competencies of individual medical and/or nursing professionals. Reports on the assessment of technical, clinical, medication and disclosure skills were excluded.

Results

Thirty-four assessment tools in 48 studies were identified: 20 tools for medical professionals, nine tools for nursing professionals, and five tools for both medical and nursing professionals. Twenty of these tools assessed the two highest Miller levels (‘shows how’ and ‘does’) and four tools were directed at multiple levels. Most of the tools that aimed at the higher levels assessed the skills of working in teams (17 tools), risk management (15 tools), and communication (11 tools). Internal structure (reliability, 22 tools) and content validity (14 tools) when described were found to be moderate. Only a small number of tools addressed the relationship between the tool itself and (1) other assessments (concurrent, predictive validity, eight tools), and (2) educational outcomes (seven tools).

Conclusions

There are many tools designed to assess the safety competencies of healthcare professionals. However, a reliable and valid toolbox for summative testing that covers all patient safety domains at Miller's four competency levels cannot yet be constructed. Many tools, however, are useful for formative feedback.

Anaesthetists' non-technical skills

This review presents the background to the development of the anaesthetists' non-technical skills (ANTS) taxonomy and behaviour rating tool, which is the first non-technical skills framework specifically designed for anaesthetists. We share the experience of the anaesthetists who designed ANTS in relation to applying it in a department of anaesthesia, using it in a simulation centre, and the process of introducing it to the profession on a national basis. We also consider how ANTS is being applied in relation to training and research in other countries and finally, we discuss emerging issues in relation to the introduction of a non-technical skills approach in anaesthesia.