1
|
Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM. Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet. ERGONOMICS 2024; 67:695-715. [PMID: 37523211 DOI: 10.1080/00140139.2023.2240048] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 07/13/2023] [Indexed: 08/01/2023]
Abstract
Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.
Collapse
Affiliation(s)
- Adam Hulme
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Southern Queensland Rural Health (SQRH), Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, Australia
| | - Neville A Stanton
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Transportation Research Group, Faculty of Engineering and Physical Sciences, University of Southampton, Southampton, UK
| | - Guy H Walker
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Centre for Sustainable Road Freight, School of Energy, Geoscience, Infrastructure and Society, Heriot-Watt University, Scotland, UK
| | - Patrick Waterson
- Human Factors and Complex Systems Group, Design School, Loughborough University, Leicestershire, UK
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
| |
Collapse
|
2
|
Baber C, Young MS. Making ergonomics accountable: Reliability, validity and utility in ergonomics methods. APPLIED ERGONOMICS 2022; 98:103583. [PMID: 34634582 DOI: 10.1016/j.apergo.2021.103583] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Revised: 08/16/2021] [Accepted: 09/07/2021] [Indexed: 06/13/2023]
Abstract
In this paper, we discuss the ways in which Neville Stanton has challenged himself, his research colleagues, PhD students, the many co-authors and contributors to his publications, and the entire Ergonomics community to determine what it means for there to be 'consistent standards for how [Ergonomics] methods are described and reported.' Only in this way, can it be possible to make claims about whether or not a method in Ergonomics is effective. Given that he is Chartered as both an Occupational Psychologist and an Ergonomist, it is not surprising that he has been concerned with the question of the reliability and validity of Ergonomics methods. In Occupational Psychology, psychometric and personnel selection methods are expected to exhibit acceptable levels of reliability, but this is an expectation which is still somewhat alien to Ergonomics. Neville's work has been instrumental in raising this issue and in providing approaches which can be used to critically evaluate the methods we use. We think that, despite his ground-breaking work, there is still much to do in the Ergonomics community to create the situation for which he has long argued.
Collapse
Affiliation(s)
- Chris Baber
- School of Computer Science, University of Birmingham, Birmingham, West Midlands, B15 2TT, UK.
| | - Mark S Young
- Human Factors and Complex Systems Group, School of Design and Creative Arts, Loughborough University, Loughborough, Leicestershire, LE11 3TU, UK.
| |
Collapse
|
3
|
Hulme A, Thompson J, Plant KL, Read GJM, Mclean S, Clacy A, Salmon PM. Applying systems ergonomics methods in sport: A systematic review. APPLIED ERGONOMICS 2019; 80:214-225. [PMID: 29674008 DOI: 10.1016/j.apergo.2018.03.019] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2017] [Revised: 01/29/2018] [Accepted: 03/31/2018] [Indexed: 06/08/2023]
Abstract
INTRODUCTION As sports systems become increasingly more complex, competitive, and technology-centric, there is a greater need for systems ergonomics methods to consider the performance, health, and safety of athletes in context with the wider settings in which they operate. Therefore, the purpose of this systematic review was to identify and critically evaluate studies which have applied a systems ergonomics research approach in the context of sports performance and injury management. MATERIAL AND METHODS Five databases (PubMed, Scopus, ScienceDirect, Web of Science, and SPORTDiscus) were searched for the dates 01 January 1990 to 01 August 2017, inclusive, for original peer-reviewed journal articles and conference papers. Reported analyses were underpinned by a recognised systems ergonomics method, and study aims were related to the optimisation of sports performance (e.g. communication, playing style, technique, tactics, or equipment), and/or the management of sports injury (i.e. identification, prevention, or treatment). RESULTS A total of seven articles were identified. Two articles were focussed on understanding and optimising sports performance, whereas five examined sports injury management. The methods used were the Event Analysis of Systemic Teamwork, Cognitive Work Analysis (the Work Domain Analysis Abstraction Hierarchy), Rasmussen's Risk Management Framework, and the Systems Theoretic Accident Model and Processes method. The individual sport application was distance running, whereas the team sports contexts examined were cycling, football, Australian Football League, and rugby union. CONCLUSIONS The included systems ergonomics applications were highly flexible, covering both amateur and elite sports contexts. The studies were rated as valuable, providing descriptions of injury controls and causation, the factors influencing injury management, the allocation of responsibilities for injury prevention, as well as the factors and their interactions underpinning sports performance. Implications and future directions for research are described.
Collapse
Affiliation(s)
- Adam Hulme
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia.
| | - Jason Thompson
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia; Transport, Health and Urban Design (THUD) Research Hub, Faculty of Architecture, Building and Planning, Melbourne School of Design, The University of Melbourne, Melbourne, Victoria, 3010, Australia
| | - Katherine L Plant
- Transportation Research Group, Faculty of Engineering and the Environment, University of Southampton, Southampton, SO17 1BJ, United Kingdom
| | - Gemma J M Read
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Scott Mclean
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Amanda Clacy
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| |
Collapse
|
4
|
Lau N, Jamieson GA, Skraaning G. Situation awareness acquired from monitoring process plants - the Process Overview concept and measure. ERGONOMICS 2016; 59:976-988. [PMID: 26653273 DOI: 10.1080/00140139.2015.1100329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
We introduce Process Overview, a situation awareness characterisation of the knowledge derived from monitoring process plants. Process Overview is based on observational studies of process control work in the literature. The characterisation is applied to develop a query-based measure called the Process Overview Measure. The goal of the measure is to improve coupling between situation and awareness according to process plant properties and operator cognitive work. A companion article presents the empirical evaluation of the Process Overview Measure in a realistic process control setting. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA based on data collected by process experts. Practitioner Summary: The Process Overview Measure is a query-based measure for assessing operator situation awareness from monitoring process plants in representative settings.
Collapse
Affiliation(s)
- Nathan Lau
- a Grado Department of Industrial & Systems Engineering , VirginiaTech , Blacksburg, VA , USA
| | - Greg A Jamieson
- b Department of Mechanical and Industrial Engineering , University of Toronto , Toronto , Canada
| | - Gyrd Skraaning
- c Industrial Psychology , OECD Halden Reactor Project , Halden , Norway
| |
Collapse
|
5
|
Neville TJ, Salmon PM. Never blame the umpire - a review of Situation Awareness models and methods for examining the performance of officials in sport. ERGONOMICS 2016; 59:962-975. [PMID: 26647641 DOI: 10.1080/00140139.2015.1100758] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
As sport becomes more complex, there is potential for ergonomics concepts to help enhance the performance of sports officials. The concept of Situation Awareness (SA) appears pertinent given the requirement for officials to understand what is going on in order to make decisions. Although numerous models exist, none have been applied to examine officials, and only several recent examples have been applied to sport. This paper examines SA models and methods to identify if any have applicability to officials in sport (OiS). Evaluation of the models and methods identified potential applications of individual, team and systems models of SA. The paper further demonstrates that the Distributed Situation Awareness model is suitable for studying officials in fastball sports. It is concluded that the study of SA represents a key area of multidisciplinary research for both ergonomics and sports science in the context of OiS. Practitioner Summary: Despite obvious synergies, applications of cognitive ergonomics concepts in sport are sparse. This is especially so for Officials in Sport (OiS). This article presents an evaluation of Situation Awareness models and methods, providing practitioners with guidance on which are the most suitable for OiS system design and evaluation.
Collapse
Affiliation(s)
- Timothy J Neville
- a Faculty of Arts and Business , Centre for Human Factors and Sociotechnical Systems , Maroochydore , Australia
- b Joint and Operations Analysis Division , Defence Science and Technology Group , Canberra , Australia
| | - Paul M Salmon
- a Faculty of Arts and Business , Centre for Human Factors and Sociotechnical Systems , Maroochydore , Australia
| |
Collapse
|
7
|
Lau N, Jamieson GA, Skraaning G. Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants. ERGONOMICS 2016; 59:393-408. [PMID: 26398584 DOI: 10.1080/00140139.2015.1080310] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
The Process Overview Measure is a query-based measure developed to assess operator situation awareness (SA) from monitoring process plants. A companion paper describes how the measure has been developed according to process plant properties and operator cognitive work. The Process Overview Measure demonstrated practicality, sensitivity, validity and reliability in two full-scope simulator experiments investigating dramatically different operational concepts. Practicality was assessed based on qualitative feedback of participants and researchers. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA in full-scope simulator settings based on data collected on process experts. Thus, full-scope simulator studies can employ the Process Overview Measure to reveal the impact of new control room technology and operational concepts on monitoring process plants. Practitioner Summary: The Process Overview Measure is a query-based measure that demonstrated practicality, sensitivity, validity and reliability for assessing operator situation awareness (SA) from monitoring process plants in representative settings.
Collapse
Affiliation(s)
- Nathan Lau
- a Grado Department of Industrial & Systems Engineering , Virginia Tech , Blacksburg , VA , USA
| | - Greg A Jamieson
- b Department of Mechanical and Industrial Engineering , University of Toronto , Toronto , Canada
| | - Gyrd Skraaning
- c Industrial Psychology, OECD Halden Reactor Project , Halden , Norway
| |
Collapse
|