Validation of a measurement tool for self-assessment of teamwork in intensive care

Trainee anaesthetist self-assessment using an entrustment scale in workplace-based assessment

The role of self-assessment in workplace-based assessment remains contested. However, anaesthesia trainees need to learn to judge the quality of their own work. Entrustment scales have facilitated a shared understanding of performance standards among supervisors by aligning assessment ratings with everyday clinical supervisory decisions. We hypothesised that if the entrustment scale similarly helped trainees in their self-assessment, there would be substantial agreement between supervisor and trainee ratings. We collected separate mini-clinical evaluation exercises forms from 113 anaesthesia trainee-supervisor pairs from three hospitals in Australia and New Zealand. We calculated the agreement between trainee and supervisor ratings using Pearson and intraclass correlation coefficients. We also tested for associations with demographic variables and examined narrative comments for factors influencing rating. We found ratings agreed in 32% of cases, with 66% of trainee ratings within one point of the supervisor rating on a nine-point scale. The correlation between trainee and supervisor ratings was 0.71, and the degree of agreement measured by the intraclass correlation coefficient was 0.67. With higher supervisor ratings, trainee ratings better correlated with supervisor ratings. We found no strong association with demographic variables. Possible explanations of divergent ratings included one party being unaware of a vital aspect of the performance and different interpretations of the prospective nature of the scale. The substantial concordance between trainee and supervisor ratings supports the contention that the entrustment scale helped produce a shared understanding of the desired performance standard. Discussion between trainees and supervisors on the reasoning underlying their respective judgements would provide further opportunities to enhance this shared understanding.

Observation of behavioural skills by medical simulation facilitators: a cross-sectional analysis of self-reported importance, difficulties, observation strategies and expertise development

BACKGROUND

The association between team performance and patient care was an immense boost for team-based education in health care. Behavioural skills are an important focus in these sessions, often provided via a mannikin-based immersive simulation experience in a (near) authentic setting. Observation of these skills by the facilitator(s) is paramount for facilitated feedback with the team. Despite the acknowledgement that trained facilitators are important for optimal learning, insight into this observation process by facilitators is limited.

OBJECTIVES

What are the self-reported current practices and difficulties regarding the observation of behavioural skills amongst facilitators during team training and how have they been trained to observe behavioural skills?

METHODS

This cross-sectional study used a pilot-tested, content-validated, multi-linguistic online survey within Europe, distributed through a non-discriminative snowball sampling method. Inclusion was limited to facilitators observing behavioural skills within a medical team setting.

RESULTS

A total of 175 persons filled in the questionnaire. All aspects of behavioural skill were perceived as very important to observe. The self-perceived difficulty of the behavioural skill aspects ranged from slightly to moderately difficult. Qualitative analysis revealed three major themes elaborating on this perceived difficulty: (1) not everything can be observed, (2) not everything is observed and (3) interpretation of observed behavioural skills is difficult. Additionally, the number of team members health care facilitators have to observe, outnumbers their self-reported maximum. Strategies and tools used to facilitate their observation were a blank notepad, co-observers and predefined learning goals. The majority of facilitators acquired observational skills through self-study and personal experience and/or observing peers. Co-observation with either peers or experts was regarded as most learn some for their expertise development. Overall, participants perceived themselves as moderately competent in the observation of behavioural skills during team training.

CONCLUSIONS

Observation of behavioural skills by facilitators in health care remains a complex and challenging task. Facilitators' limitations with respect to attention, focus and (in)ability to perform concomitant tasks, need to be acknowledged. Although strategies and tools can help to facilitate the observation process, they all have their limitations and are used in different ways.

Simulation-based training program on patient safety management: A quasi-experimental study among new intensive care unit nurses

BACKGROUND

Patient safety is an essential responsibility of nurses. However, newly graduated nurses experience patient safety incidents due to a lack of patient safety competency. In particular, intensive care unit nurses provide care to critical patients, making it difficult for new nurses to maintain patient safety. Therefore, it is necessary to improve the patient safety competency of newly graduated nurses working in the intensive care unit.

OBJECTIVES

To analyze the effects of a simulation-based training program on patient safety management on the patient safety competency, patient safety management activity, communication self-efficacy, and teamwork of newly graduated nurses working in an intensive care unit.

DESIGN

A quasi-experimental study.

SETTINGS

The study was conducted at a university in Korea.

PARTICIPANTS

A total of 44 newly graduated nurses working in the intensive care unit of a university hospital.

METHODS

A simulation-based training program on patient safety management was applied to the experimental group (n = 22), while only the usual hospital training program was used for the control group (n = 22). A structured questionnaire survey comprising Patient Safety Competency Self-Evaluation, Patient Safety Management Activities, the Korean version of the self-efficacy questionnaire, and K-self assessment of teamwork was conducted at baseline test (T1), post test (T2), and follow-up test (T3). Data were analyzed using a repeated measures ANOVA.

RESULTS

The experimental group showed significantly higher patient safety competency, patient safety knowledge, and patient safety skills, along with higher communication self-efficacy scores than the control group (p < .05) after the intervention and at four weeks after the intervention.

CONCLUSIONS

The simulation-based training program on patient safety management effectively improved the patient safety competency, patient safety knowledge, and patient safety skills as well as communication self-efficacy of newly graduated nurses working in an intensive care unit.

Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators' self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Strengthening teamwork capability in allied health: implementation of a team development program in a metropolitan health service

Objective Collaborative practice is critical to optimising patient outcomes in contemporary healthcare settings. Evidence suggests interprofessional learning is an effective way to develop teamwork capabilities, yet these skills are traditionally developed in professional silos, or not at all. This study evaluated the implementation of a team development program, the Team Management Systems (TMS) program, for allied health staff within a large metropolitan health service. Methods A mixed-methods audit-quality improvement study was conducted, using Kirkpatrick's four-level evaluation model to structure evaluation of the program. Semistructured questionnaire and workforce survey data were retrieved immediately, 6 months and 1-2 years after training and applied to each level of the model (Reaction, Learning, Behaviour, Results). Results In all, 886 staff participated in the TMS program from 2014 to 2018. High satisfaction with the program was observed. Knowledge of what constitutes effective teamwork improved significantly (P=0.008) in TMS participants compared with a matched untrained cohort. Participants reported positive behaviour change and continued engagement with TMS principles 6 months after training. Perceived impact of the program on patient and/or organisational outcomes was evident, although less compelling than the changes to knowledge and behaviour. Conclusions The TMS program yielded positive effects on staff satisfaction, knowledge, team dynamics and team behaviours. These findings demonstrate the significant value of such initiatives to enhance the capability and effectiveness of interdisciplinary healthcare teams. What is known about the topic? Complex conditions, increasing comorbidities, specialisation and scarcity of resources mean healthcare workers need to work effectively in teams to achieve quality, safe, person centred patient care. There is some evidence of the effect of teamwork initiatives on knowledge or behaviour in specific clinical specialities, single services or single professions, but limited research is available regarding the effects of teamwork programs across multiple professions, including allied health professions, and on patient and organisational outcomes. What does this paper add? This paper describes the effect of a large-scale teamwork program implemented across multiple professions, including enablers and barriers. It presents outcomes at all four levels of Kirkpatrick's evaluation model, including the less studied behaviour and results levels. What are the implications for practitioners? This paper supports health service leaders to consider developing and implementing interprofessional teamwork programs to foster essential teamwork capabilities. Learning together about teamwork, across professional silos, will lead to collaborative, patient-centred care, which leads to safe, quality patient outcomes.

Improving the Quality of Evaluation Data in Simulation-Based Healthcare Improvement Projects: A Practitioner's Guide to Choosing and Using Published Measurement Tools

STATEMENT

Simulation is increasingly being used in healthcare improvement projects. The aims of such projects can be extremely diverse. Accordingly, the outcomes or participant attributes that need to be measured can vary dramatically from project-to-project and may include a wide range of nontechnical skills, technical skills, and psychological constructs. Consequently, there is a growing need for simulation practitioners to be able to identify suitable measurement tools and incorporate them into their work. This article provides a practical introduction and guide to the key considerations for practitioners when selecting and using such tools. It also offers a substantial selection of example tools, both to illustrate the key considerations in relation to choosing a measure (including reliability and validity) and to serve as a convenient resource for those planning a study. By making well-informed choices, practitioners can improve the quality of the data they collect, and the likelihood that their projects will succeed.

Systematic review of the characteristics of brief team interventions to clarify roles and improve functioning in healthcare teams

AIM

Describe brief (less than half a day) interventions aimed at improving healthcare team functioning.

METHODS

A systematic review on brief team interventions aimed at role clarification and team functioning (PROSPERO Registration Number: CRD42018088922). Experimental or quasi-experimental studies were included. Database searches included CINAHL, Medline, EMBASE, PUBMED, Cochrane, RCT Registry-1990 to April 2020 and grey literature. Articles were screened independently by teams of two reviewers. Risk of bias was assessed. Data from the retained articles were extracted by one reviewer and checked by a second reviewer independently. A narrative synthesis was undertaken.

RESULTS

Searches yielded 1928 unique records. Final sample contained twenty papers describing 19 studies, published between 2009 and 2020. Studies described brief training interventions conducted in acute care in-patient settings and included a total of 6338 participants. Participants' socio-demographic information was not routinely reported. Studies met between two to six of the eight risk of bias criteria. Interventions included simulations for technical skills, structured communications and speaking up for non-technical skills and debriefing. Debriefing sessions generally lasted between five to 10 minutes. Debriefing sessions reflected key content areas but it was not always possible to determine the influence of the debriefing session on participants' learning because of the limited information reported.

DISCUSSION

Interest in short team interventions is recent. Single two-hour sessions appear to improve technical skills. Three to four 30- to 60-minute training sessions spread out over several weeks with structured facilitation and debriefing appear to improve non-technical skills. Monthly meetings appear to sustain change over time.

CONCLUSION

Short team interventions show promise to improve team functioning. Effectiveness of interventions in primary care and the inclusion of patients and families needs to be examined. Primary care teams are structured differently than teams in acute care and they may have different priorities.

Exploring the non-technical competencies for on-scene public health responders in chemical, biological, radiological, and nuclear emergencies: a qualitative study

Objectives

The purpose of this study was to define and delineate specific non-technical competencies for first-line public health responders in Chemical, Biological, Radiological, and Nuclear (CBRN) emergencies in China.

Study design

A qualitative study was conducted in China involving interviews with key informants in the field of health response to CBRN disasters.

Methods

One-on-one in-depth interviews were carried out with 20 participants, including expert members of National Medical Response Teams for CBRN disasters, officials at emergency management authorities, and scholars of academic institutions related to CBRN emergency. Interviews were recorded using audio equipment, transcribed, and coded into codable passages as per grounded theory using NVivo software. Themes were identified within the transcriptions by using thematic analysis.

Results

A total of 159 codable passages were produced. Eight domains of non-technical core competencies were identified: (1) situation awareness, (2) communication skills, (3) collaboration, (4) resource management, (5) task management, (6) cultural competency, (7) austere environment skills, and (8) physical stamina.

Conclusions

The study identified a variety of competencies for on-scene public health responders in CBRN emergencies. The findings of this study could specifically benefit development of strategy and improvement of content of education and training. Further research that involves input from the disaster response community at large is needed for the validation of these competencies.

•

Non-technical competencies affect performance of health response to Chemical, Biological, Radiological, and Nuclear (CBRN) disasters.

•

Austere environment skills and physical stamina are priority non-technical competencies for on-site public health responders in CBRN events.

•

Effective CBRN health emergency training should integrate and synchronize courses based on non-technical/technical competency.

Nontechnical Competency Framework for Health Professionals in All-Hazard Emergency Environment: A Systematic Review

OBJECTIVES

To summarize characteristics and commonalities of non-technical competency frameworks for health professionals in emergency and disaster.

METHODS

An electronic literature search was conducted in PubMed, MEDLINE, ERIC, Scopus, Cochrane database, and Google Scholar to identify original English-language articles related to development, evaluation or application of the nontechnical competency frameworks. Reviewers assessed identified articles for exclusion/inclusion criteria and abstracted data on study design, framework characteristics, and reliability/validity evidence.

RESULTS

Of the 9627 abstracts screened, 65 frameworks were identified from 94 studies that were eligible for result extraction. Sixty (63.8%) studies concentrated on clinical settings. Common scenarios of the studies were acute critical events in hospitals (44;46.8%) and nonspecified disasters (39;41.5%). Most of the participants (76; 80.9%) were clinical practitioners, and participants in 36 (38.3%) studies were multispecialty. Thirty-three (50.8%) and 42 (64.6%) frameworks had not reported evidence on reliability and validity, respectively. Fourteen of the most commonly involved domains were identified from the frameworks.

CONCLUSIONS

Nontechnical competency frameworks applied to multidisciplinary emergency health professionals are heterogeneous in construct and application. A fundamental framework with standardized terminology for the articulation of competency should be developed and validated so as to be accepted and adapted universally by health professionals in all-hazard emergency environment.

Effects of a simulation-based education program for nursing students responding to mass casualty incidents: A pre-post intervention study

BACKGROUND

The necessity of disaster preparedness among nursing students has been continuously emphasized.

OBJECTIVES

This study aimed to verify the effectiveness of a simulation-based education program for nursing students responding to mass casualty incidents (MCI) from the perspectives of triage accuracy, response attitude, teamwork, and program satisfaction.

DESIGN

This study employed a pre-post intervention design.

SETTINGS

Disaster Simulation Lab and a debriefing room in the University Nursing Simulation Center in South Korea.

PARTICIPANTS

The participants were 34 graduating nursing students attending a university in Seoul.

METHODS

The program consisted of lectures on disaster nursing, group discussions, practice, debriefings, and a pre- and post-test, conducted over 180 min. Simulation-based training was conducted using the Emergo Train System®. The simulation environment comprised pre-hospital and hospital sections, with videos displayed on a large screen and sound effects played on loudspeakers.

RESULTS

Participants were likely to undertriage. There was a significant increase in positive attitudes after the intervention (p < .001). Self-reported teamwork was high, and among its subfactors, "leadership and team coordination" scored the highest. Participants' satisfaction with the program was high (4.5/5.0).

CONCLUSIONS

The simulation-based MCI program was effective in boosting positive attitudes among nursing students. In future, comparative studies including control groups and different instructional methods should be conducted. A patient bank should also be developed considering participants' knowledge levels and the circumstances of each country.

Workload and influencing factors in non-emergency medical transfers: a multiple linear regression analysis of a cross-sectional questionnaire study

Background

Human workload is a key factor for system performance, but data on emergency medical services (EMS) are scarce. We investigated paramedics’ workload and the influencing factors for non-emergency medical transfers. These missions make up a major part of EMS activities in Germany and are growing steadily in number.

Methods

Paramedics rated missions retrospectively through an online questionnaire. We used the NASA-Task Load Index (TLX) to quantify workload and asked about a variety of medical and procedural aspects for each mission. Teamwork was assessed by the Weller teamwork measurement tool (TMT). With a multiple linear regression model, we identified a set of factors leading to relevant increases or decreases in workload.

Results

A total of 194 non-emergency missions were analysed. Global workload was rated low (Mean = 27/100). In summary, 42.8% of missions were rated with a TLX under 20/100. TLX subscales revealed low task demands but a very positive self-perception of performance (Mean = 15/100). Teamwork gained high ratings (Mean TMT = 5.8/7), and good teamwork led to decreases in workload. Aggression events originating from patients and bystanders occurred frequently (n = 25, 12.9%) and increased workload significantly. Other factors affecting workload were the patient’s body weight and the transfer of patients with transmittable pathogens.

Conclusion

The workload during non-emergency medical transfers was low to very low, but performance perception was very positive, and no indicators of task underload were found. We identified several factors that led to workload increases. Future measures should attempt to better train paramedics for aggression incidents, to explore the usefulness of further technical aids in the transfer of obese patients and to reconsider standard operating procedures for missions with transmittable pathogens.

Evaluation of teamwork assessment tools for interprofessional simulation: a systematic literature review

There is growing evidence supporting the use of simulation-based education to improve teamwork in the clinical environment, which results in improved patient outcomes. Interprofessional simulation improves awareness of professional roles and responsibilities, promotes teamwork and provides training in non-technical skills. Tools have been developed to assess the quality of teamwork during simulation, but the use of these tools should be supported by validity evidence in appropriate contexts. This study aims to assess the validity of teamwork tools used in simulation-based interprofessional training for healthcare workers and students, and to compare the design and reporting of these studies. Medline, EMBASE, ERIC, and CINAHL were searched using terms synonymous with simulation, crew resource management, training, assessment, interprofessional, and teamwork, from 2007-2017. Interprofessional healthcare simulation studies involving objectively rated teamwork training were included. The initial search provided 356 records for review, of which 24 were ultimately included. Three tools demonstrated good validity evidence underpinning their use. However, three studies did not explore tool psychometrics at all, and the quality of reporting amongst these studies on design and participant demographics was variable. Further research to generate reporting guidelines and validate existing tools for new populations would be beneficial.

Single mission workload and influencing factors in German prehospital emergency medicine - a nationwide prospective survey of 1361emergency missions

Background

Workload is a major determinant of system performance and human well-being. This study aims to evaluate workload in prehospital emergency medicine on a single mission level and investigates influencing factors originating from medical scenarios, patient-provider interaction, EMS logistics and teamwork.

Methods

In a nationwide study, German paramedics were asked to evaluate single missions for perceived workload by completing the NASA Task-Load-Index (TLX). A variety of candidate variables were documented and tested for influence on the TLX through multivariate regression analysis.

Results

One thousand three hundred sixty-one emergency missions were analysed. Global workload scored in medium ranges (Median TLX 41.00/100; IQR 24.25–57.50). 263 missions achieved very low (< 20/100) and 52 missions achieved very high (> 80/100) levels of workload. Severity of distress as indicated by the NACA score (delta TLX 2.71 per 1 NACA point), execution of invasive procedures (e.g. delta TLX 8.20 for intravenous access), obese patients (delta TLX 0.05 per 1 kg of weight) and aggression incidences (e.g. delta TLX 10.54 for physical aggression), amongst others, resulted in significant increases in workload. Good teamwork decreased workload by 2.18 points per 1 point on the Weller-Teamwork Measurement Tool.

Conclusion

Distinct factors result in significant increases in workload for EMS paramedics. Improvements in training for certain medical scenarios, strategies against aggression events and enhancements in EMS logistics - especially for the transfer of obese patients – should be implemented and tested for their presumably positive effect on workload, EMS performance and paramedics’ well-being.

Electronic supplementary material

The online version of this article (10.1186/s13049-019-0650-2) contains supplementary material, which is available to authorized users.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Monitoring teamwork: a narrative review

A narrative review was carried out to identify articles on monitoring of teamwork, with particular relevance to anaesthetists. The papers reviewed showed that team monitoring takes place both implicitly and explicitly in the anaesthetic environment. No single optimal model of teamwork monitoring for all situations was identified. Most of the studies identified were of a pre-intervention, post-intervention design, without randomisation or control group. Information shared during a formal briefing is more likely to be recalled, and provides a basis for a shared team mental model. A number of studies appeared to show that targeted teamwork training has a positive impact on both teamwork and patient safety.

Validation of the self-assessment teamwork tool (SATT) in a cohort of nursing and medical students

INTRODUCTION

Poor teamwork has been implicated in medical error and teamwork training has been shown to improve patient care. Simulation is an effective educational method for teamwork training. Post-simulation reflection aims to promote learning and we have previously developed a self-assessment teamwork tool (SATT) for health students to measure teamwork performance. This study aimed to evaluate the psychometric properties of a revised self-assessment teamwork tool.

METHODS

The tool was tested in 257 medical and nursing students after their participation in one of several mass casualty simulations.

RESULTS

Using exploratory and confirmatory factor analysis, the revised self-assessment teamwork tool was shown to have strong construct validity, high reliability, and the construct demonstrated invariance across groups (Medicine & Nursing).

CONCLUSIONS

The modified SATT was shown to be a reliable and valid student self-assessment tool. The SATT is a quick and practical method of guiding students' reflection on important teamwork skills.

Development of a self-assessment teamwork tool for use by medical and nursing students

BACKGROUND

Teamwork training is an essential component of health professional student education. A valid and reliable teamwork self-assessment tool could assist students to identify desirable teamwork behaviours with the potential to promote learning about effective teamwork. The aim of this study was to develop and evaluate a self-assessment teamwork tool for health professional students for use in the context of emergency response to a mass casualty.

METHODS

The authors modified a previously published teamwork instrument designed for experienced critical care teams for use with medical and nursing students involved in mass casualty simulations. The 17-item questionnaire was administered to students immediately following the simulations. These scores were used to explore the psychometric properties of the tool, using Exploratory and Confirmatory Factor Analysis.

RESULTS

202 (128 medical and 74 nursing) students completed the self-assessment teamwork tool for students. Exploratory factor analysis revealed 2 factors (5 items - Teamwork coordination and communication; 4 items - Information sharing and support) and these were justified with confirmatory factor analysis. Internal consistency was 0.823 for Teamwork coordination and communication, and 0.812 for Information sharing and support.

CONCLUSIONS

These data provide evidence to support the validity and reliability of the self-assessment teamwork tool for students This self-assessment tool could be of value to health professional students following team training activities to help them identify the attributes of effective teamwork.