Cognitive work analysis: An influential legacy extending beyond human factors and engineering

Naturalistic observations of multiteam interaction networks: Implications for cognition in crisis management teams

Interaction has been recognised as an essential lens to understand how cognition is formed in a complex adaptive team such as a multidisciplinary crisis management team (CMT). However, little is known about how interactions within and across CMTs give rise to the multi-team system's overall cognitive functioning, which is essential to avoid breakdowns in coordination. To address this gap, we characterise and compare the component CMTs' role-as-intended (RAI) and role-as-observed (RAO) in adapting to the complexity of managing informational needs. To characterise RAI, we conducted semi-structured interviews with subject matter experts and then made a qualitative synthesis using a thematic analysis method. To characterise RAO, we observed multiteam interaction networks in real-time at a simulated training environment and then analysed the component CMTs' relative importance using node centrality measures. The resulting inconsistencies between RAI and RAO imply the need to investigate cognition in multiple CMTs through the lens of interaction.Practitioner summary: When a disaster occurs, multidisciplinary CMTs are expected to serve their roles as described in written or verbal guidelines. However, according to our naturalistic observations of multiteam interaction networks, such descriptions may be (necessary but) insufficient for designing, training, and evaluating CMTs in the complexity of managing informational needs together.

Designing human-AI systems for complex settings: ideas from distributed, joint, and self-organising perspectives of sociotechnical systems and cognitive work analysis

Real-world events like the COVID-19 pandemic and wildfires in Australia, Europe, and America remind us that the demands of complex operational settings are met by multiple, distributed teams interwoven with a large array of artefacts and networked technologies, including automation. Yet, current models of human-automation interaction, including those intended for human-machine teaming or collaboration, tend to be dyadic in nature, assuming individual humans interacting with individual machines. Given the opportunities and challenges of emerging artificial intelligence (AI) technologies, and the growing interest of many organisations in utilising these technologies in complex operations, we suggest turning to contemporary perspectives of sociotechnical systems for a way forward. We show how ideas of distributed cognition, joint cognitive systems, and self-organisation lead to specific concepts for designing human-AI systems, and propose that design frameworks informed by contemporary views of complex work performance are needed. We discuss cognitive work analysis as an example.

Systems thinking and designerly tools for medical device design in engineering curricula

Background

In this paper we focus on medical device development (MDD) in Industrial Design Engineering (IDE) academia. We want to find which methods our MDD-students currently use, where our guidance has shortcomings and where it brings added value.

Methods

We have analysed 19 master and 3 doctoral MDD-theses in our IDE curriculum. The evaluation focusses around four main themes: 1) regulatory 2) testing 3) patient-centricity and 4) systemic design.

Results

Regulatory aspects and medical testing procedures seem to be disregarded frequently. We assume this is because of a lack of MDD experience and the small thesis timeframe. Furthermore, many students applied medical-oriented systemic tools, which enhances multiperspectivism. However, we found an important lack in the translation to the List of Specifications and to business models of these medical devices. Finally, students introduced various participatory techniques, but seem to struggle with implementing this in the setting of evidence-based medicine.

Multiple outcome interactions in healthcare systems: a participatory outcome mapping approach

Outcomes, which are the result state or condition from a process or intervention, are essential elements of healthcare system design and an important indicator of performance. They are included in well-known system analysis frameworks such as the Systems Engineering Initiative for Patient Safety (SEIPS) and Cognitive Work Analysis (CWA). However, fewer practical approaches exist for understanding and communicating interactions among healthcare outcomes. This study applies a novel mapping method as a practical approach to collect, aggregate and visualise interrelations among multiple healthcare outcomes. Graphic facilitation mapping sessions with eleven healthcare providers and ten patients with chronic conditions were conducted. Participants created outcome interrelationship maps following a six-step process. Two outcome-based network visualisations were synthesised using network analysis. This outcome-based approach advances how we frame healthcare systems, focussing on accommodating multiple stakeholders' visions, understanding interrelations, and defining trade-offs. This practical approach may complement frameworks such as SEIPS and CWA. Practitioner summary: The presented outcome-based mapping approach can facilitate the understanding of outcomes as part of the interrelated healthcare system. The approach allows the discussion and integration of different stakeholders' outcome priorities to identify critical elements and better inform the development or adaptation of healthcare systems. Abbreviations: SEIPS: engineering initiative for patient safety; CWA: cognitive work analysis; HFE: human factors and ergonomics; AH: abstraction hierarchy; POTS: postural orthostatic tachycardia syndrome; DRM: design research methodology; FA2: forceAtlas2; HbA1c: glycated haemoglobin; NHS: National Health Service; UK: United Kingdom.

Human Autonomy in Future Drone Traffic: Joint Human-AI Control in Temporal Cognitive Work

The roles of human operators are changing due to increased intelligence and autonomy of computer systems. Humans will interact with systems at a more overarching level or only in specific situations. This involves learning new practices and changing habitual ways of thinking and acting, including reconsidering human autonomy in relation to autonomous systems. This paper describes a design case of a future autonomous management system for drone traffic in cities in a key scenario we call The Computer in Brussels. Our approach to designing for human collaboration with autonomous systems builds on scenario-based design and cognitive work analysis facilitated by computer simulations. We use a temporal method, called the Joint Control Framework to describe human and automated work in an abstraction hierarchy labeled Levels of Autonomy in Cognitive Control. We use the Score notation to analyze patterns of temporal developments that span levels of the abstraction hierarchy and discuss implications for human-automation communication in traffic management. We discuss how autonomy at a lower level can prevent autonomy on higher levels, and vice versa. We also discuss the temporal nature of autonomy in minute-to-minute operative work. Our conclusion is that human autonomy in relation to autonomous systems is based on fundamental trade-offs between technological opportunities to automate and values of what human actors find meaningful.

Determining the prerequisites for effective workplace inspection by the occupational safety and health regulatory authority using cognitive work analysis

Objectives. Earlier attempts to understand inspection work and improve inspection effectiveness are based on how controls are conducted and the interactions between the inspectors and inspection subjects. This study aimed to determine workplace occupational safety and health inspection effectiveness prerequisites using cognitive work analysis, an approach for design and evaluation of work domains, focusing on activities and work constraints. Methods. Data were collected through semi-structured interviews and a survey with labour inspection authority inspectors, and by reviewing inspection reports and earlier studies on workplace inspections. These were used in the first three cognitive work analysis phases to identify the prerequisites of effective workplace inspection and designing inspection strategies. Results. An abstraction hierarchy showing the affordances was prepared, with purpose-related functions identified as the inspection effectiveness prerequisites. A contextual activities template and a decision ladder for inspection work were prepared. Strategy maps for on-site control were created, allowing design of structured and organized workplace inspection strategies supporting the work domain's purposes. Conclusion. The analysis dimensions served the study sufficiently, providing the purpose-related functions with their respective subgoals and subsidiary functions that provided the prerequisite for effective workplace inspections and allowed for designing structured and organized strategies for on-site workplace inspection.

Designing for Adaptation in Workers' Individual Behaviors and Collective Structures With Cognitive Work Analysis: Case Study of the Diagram of Work Organization Possibilities

OBJECTIVE

We demonstrate that the diagram of work organization possibilities, a recent addition to cognitive work analysis, can be used to develop designs that promote adaptation in the workplace.

BACKGROUND

Workers in sociotechnical systems adapt not just their individual behaviors but also their collective structures in dealing with instability, uncertainty, and unpredictability in their tasks. However, conventional design approaches are limited in supporting adaptations in both workers' behaviors and structures, especially during unforeseen situations. The work organization possibilities diagram has the potential to meet these requirements, but its value for design has not been established.

METHOD

We present a case study of a future system for maritime surveillance that provides an analytical demonstration of the utility of the diagram for design and empirical validation of the impact, uniqueness, and feasibility of this approach in an industrial setting.

RESULTS

This application results in a team design that is integrated with the career and training progression pathway of the crew in a way that maximizes the system's behavioral and structural possibilities for adaptation. Further, the approach has impact on practice, makes a distinct contribution to design relative to other techniques, and is implemented feasibly in an industrial setting.

CONCLUSION

The work organization possibilities diagram can contribute to the development of an integrated system design that supports actors' possibilities for behavioral and structural adaptation in a unified fashion.

APPLICATION

This research provides a basis for designing interfaces, teams, training, and automation that preserve a system's inherent capacity for adaptation.

Combining cognitive work analysis and empirical evaluations to understand map use by operators of small carry-on unmanned aerial systems

Operating a small carry-on unmanned aerial system (UAS) alone is challenging. Research on facilitating single-operator work has focused mainly on payload operation and health monitoring. Little focus has been given to mission-related aspects and how the command and control (C2) map display contributes to mission accomplishment. This study uses cognitive work analysis (CWA) to describe the operational work of the mission operator of a Skylark miniature UAS system. Three CWA phases were conducted - work domain analysis, control task analysis and strategy analysis - providing a rich framework of operational mission phases, task components, processes and the physical interface-objects in use. These representations highlight the operators' extensive use of the C2 map during all mission phases, for all object-related processes. To further enhance the outcomes of the CWA, and prior to outlining specific design requirements, an empirical investigation was conducted in which the eye movements of five experienced operators were obtained during a simulated mission. The empirical results confirm and further specify the work patterns that operators adopt. Quantitative analysis shows operators' extensive focus on the map, especially during mission-critical phases. These analyses led to the conclusion that a significant change in the way operators interact with the C2 map, or alternative designs to enhance map-based information utilization, should be applied. Insights drawn from this analysis can be applied to other aerial surveillance work domains, and adding empirical evaluations is helpful to further refine and reinforce the CWA outcomes.

Opportunities and Recommendations for Improving Medication Safety: Understanding the Medication Management System in Primary Care Through an Abstraction Hierarchy

Background

Despite making great strides in improving the treatment of diseases, the minimization of unintended harm by medication therapy continues to be a major hurdle facing the health care system. Medication error and prescription of potentially inappropriate medications (PIMs) represent a prevalent source of harm to patients and are associated with increased rates of adverse events, hospitalizations, and increased health care costs. Attempts to improve medication management systems in primary care have had mixed results. Implementation of new interventions is difficult because of complex contextual factors within the health care system. Abstraction hierarchy (AH), the first step in cognitive work analysis (CWA), is used by human factors practitioners to describe complex sociotechnical systems. Although initially intended for the nuclear power domain and interface design, AH has been used successfully to aid the redesign of numerous health care systems such as the design of decision support tools, mobile patient monitoring apps, and a telephone triage system.

Objective

This paper aims to refine our understanding of the primary care office in relation to a patient’s medication through the development of an AH. Emphasis was placed on the elements related to medication safety to provide guidance for the design of a safer medication management system in primary care.

Methods

The AH development was guided by the methodology used by seminal CWA literature. It was initially developed by 2 authors and later fine-tuned by an expert panel of clinicians, social scientists, and a human factors engineer. It was subsequently refined until an agreement was reached. A means-ends analysis was performed and described for the nodes of interest. The model represents the primary care office space through functional purposes, values and priorities, function-related purposes, object-related processes, and physical objects.

Results

This model depicts the medication management system at various levels of abstraction. The resulting components must be balanced and coordinated to provide medical treatment with limited health care resources. Understanding the physical and informational constraints on activities that occur in a primary care office depicted in the AH defines areas in which medication safety can be improved.

Conclusions

Numerous means-ends relationships were identified and analyzed. These can be further evaluated depending on the specific needs of the user. Recommendations for optimizing a medication management system in a primary care facility were made. Individual practices can use AH for clinical redesign to improve prescribing and deprescribing practices.

Radical systems thinking and the future role of computational modelling in ergonomics: an exploration of agent-based modelling

We are teetering on the precipice of the imminent Fourth Industrial Revolution. In this new age, systems are set to become more densely intraconnected and interconnected, and massive sociotechnical systems exhibiting unprecedented levels of complexity will increasingly take hold. At the dawning of this new age, the Ergonomics discipline must reflect on its preparedness for tackling problems in these novel systems. This paper engages in this reflection by putting forth a critical commentary on the implication of these changes on the discipline and discusses the utility of our current methods in this new paradigm. A resulting Radical Systems Thinking in Ergonomics Manifesto is put forward - a set of mandates to guide practitioners and researchers in the development of new methods capable of coping with these imminent challenges. From the manifesto are derived a series of capability requirements for future computational modelling approaches in Ergonomics. Practitioner summary: The goal of this paper was to inspire the Ergonomics community to pursue further applications involving computational modelling approaches such as Agent-Based Modelling. It presents a manifesto for the future of the discipline, and from this the capabilities that future computational modelling approaches need to possess. Abbreviations: 1IR: first industrial revolution; 2IR: second industrial revolution; 3IR: third industrial revolution; 4IR: fourth industrial revolution; ABM: agent based model; AI: artificial intelligence; AoF: allocation of function; CPA: cyber physical attack; CPS: cyber-physical system; CWA: cognitive work analysis; DDoS: distributed denial of service; EAST: event analysis of systemic teamwork; FRAM; functional resonance analysis method; HCI: human-computer interaction; HERA: human error and recovery assessment; HET: human error template; HMC: human-machine cooperation; IoT: internet of things; RSTEM: radical systems thinking in ergonomics manifesto; SAI: situated artificial intelligence; STAMP: systems theoretic accident model and processes; TRACEr: technique for the retrospective and predictive analysis of cognitive errors in air traffic control.

Socio-Cognitive Engineering of a Robotic Partner for Child's Diabetes Self-Management

Social or humanoid robots do hardly show up in “the wild,” aiming at pervasive and enduring human benefits such as child health. This paper presents a socio-cognitive engineering (SCE) methodology that guides the ongoing research & development for an evolving, longer-lasting human-robot partnership in practice. The SCE methodology has been applied in a large European project to develop a robotic partner that supports the daily diabetes management processes of children, aged between 7 and 14 years (i.e., Personal Assistant for a healthy Lifestyle, PAL). Four partnership functions were identified and worked out (joint objectives, agreements, experience sharing, and feedback & explanation) together with a common knowledge-base and interaction design for child's prolonged disease self-management. In an iterative refinement process of three cycles, these functions, knowledge base and interactions were built, integrated, tested, refined, and extended so that the PAL robot could more and more act as an effective partner for diabetes management. The SCE methodology helped to integrate into the human-agent/robot system: (a) theories, models, and methods from different scientific disciplines, (b) technologies from different fields, (c) varying diabetes management practices, and (d) last but not least, the diverse individual and context-dependent needs of the patients and caregivers. The resulting robotic partner proved to support the children on the three basic needs of the Self-Determination Theory: autonomy, competence, and relatedness. This paper presents the R&D methodology and the human-robot partnership framework for prolonged “blended” care of children with a chronic disease (children could use it up to 6 months; the robot in the hospitals and diabetes camps, and its avatar at home). It represents a new type of human-agent/robot systems with an evolving collective intelligence. The underlying ontology and design rationale can be used as foundation for further developments of long-duration human-robot partnerships “in the wild.”

Ecological Interface Design for Computer Network Defense

OBJECTIVE

A prototype ecological interface for computer network defense (CND) was developed.

BACKGROUND

Concerns about CND run high. Although there is a vast literature on CND, there is some indication that this research is not being translated into operational contexts. Part of the reason may be that CND has historically been treated as a strictly technical problem, rather than as a socio-technical problem.

METHODS

The cognitive systems engineering (CSE)/ecological interface design (EID) framework was used in the analysis and design of the prototype interface. A brief overview of CSE/EID is provided. EID principles of design (i.e., direct perception, direct manipulation and visual momentum) are described and illustrated through concrete examples from the ecological interface.

RESULTS

Key features of the ecological interface include (a) a wide variety of alternative visual displays, (b) controls that allow easy, dynamic reconfiguration of these displays, (c) visual highlighting of functionally related information across displays, (d) control mechanisms to selectively filter massive data sets, and (e) the capability for easy expansion. Cyber attacks from a well-known data set are illustrated through screen shots.

CONCLUSION

CND support needs to be developed with a triadic focus (i.e., humans interacting with technology to accomplish work) if it is to be effective. Iterative design and formal evaluation is also required. The discipline of human factors has a long tradition of success on both counts; it is time that HF became fully involved in CND.

APPLICATION

Direct application in supporting cyber analysts.

Examining Cybersecurity of Cyberphysical Systems for Critical Infrastructures Through Work Domain Analysis

OBJECTIVE

The aim of this study was to apply work domain analysis for cybersecurity assessment and design of supervisory control and data acquisition (SCADA) systems.

BACKGROUND

Adoption of information and communication technology in cyberphysical systems (CPSs) for critical infrastructures enables automated and distributed control but introduces cybersecurity risk. Many CPSs employ SCADA industrial control systems that have become the target of cyberattacks, which inflict physical damage without use of force. Given that absolute security is not feasible for complex systems, cyberintrusions that introduce unanticipated events will occur; a proper response will in turn require human adaptive ability. Therefore, analysis techniques that can support security assessment and human factors engineering are invaluable for defending CPSs.

METHOD

We conducted work domain analysis using the abstraction hierarchy (AH) to model a generic SCADA implementation to identify the functional structures and means-ends relations. We then adopted a case study approach examining the Stuxnet cyberattack by developing and integrating AHs for the uranium enrichment process, SCADA implementation, and malware to investigate the interactions between the three aspects of cybersecurity in CPSs.

RESULTS

The AHs for modeling a generic SCADA implementation and studying the Stuxnet cyberattack are useful for mapping attack vectors, identifying deficiencies in security processes and features, and evaluating proposed security solutions with respect to system objectives.

CONCLUSION

Work domain analysis is an effective analytical method for studying cybersecurity of CPSs for critical infrastructures in a psychologically relevant manner.

APPLICATION

Work domain analysis should be applied to assess cybersecurity risk and inform engineering and user interface design.

Interactive mechanism of working environments and construction behaviors with cognitive work analysis: an elevator installation case study

Unsafe behavior is a leading factor in accidents, and the working environment significantly affects behaviors. However, few studies have focused on detailed mechanisms for addressing unsafe behaviors resulting from environmental constraints. This study aims to delineate these mechanisms using cognitive work analysis (CWA) for an elevator installation case study. Elevator installation was selected for study because it involves operations at heights: falls from heights remain a major cause of construction worker mortality. This study adopts a mixed research approach based on three research methodology stages. This research deconstructs the details of the working environment, the workers' decision-making processes, the strategies chosen given environmental conditions and the conceptual model for workers' behaviors, which jointly depict environment-behavior mechanisms at length. By applying CWA to the construction industry, environmental constraints can easily be identified, and targeted engineering suggestions can be generated.

Implementation of a novel taxonomy based on cognitive work analysis in the assessment of safety performance

PURPOSE

The aim of this study was to examine how the developed taxonomy of cognitive work analysis (CWA) can be applied in combination with statistical analysis regarding different sociotechnical categories.

MATERIALS AND METHODS

This study applied a combination of quantitative and qualitative methodologies. Workers (n = 120) and managers (n = 85) in the chemical industry were asked in a questionnaire how different occupational safety and health (OSH) measures were being implemented. The exploration of the qualitative CWA taxonomy consisted of an analysis of the following topics: (a) work domain; (b) control task; (c) strategies; (d) social organization and cooperation; (e) worker competencies.

RESULTS AND CONCLUSIONS

The following hypotheses were supported - activities of the management had positive impacts on the aggregated variables: near-accident investigation and instructions (H₁); OSH training (H₂); operations, technical processes and safe use of chemicals (H₃); use of personal protective equipment (H₄); measuring, follow-up and prevention of major accidents (H₅). The CWA taxonomy was applied in mixed methods when testing H₁-H₅. A special approach is to analyze the work demands of complex sociotechnical systems with the taxonomy of CWA. In problem-solving, the CWA taxonomy should seek to capitalize on the strengths and minimize the limitations of safety performance.