Neuroergonomic Attention Assessment in Safety-Critical Tasks: EEG Indices and Subjective Metrics Validation in a Novel Task-Embedded Reaction Time Paradigm

Background/Objectives: This study addresses the gap in methodological guidelines for neuroergonomic attention assessment in safety-critical tasks, focusing on validating EEG indices, including the engagement index (EI) and beta/alpha ratio, alongside subjective ratings. Methods: A novel task-embedded reaction time paradigm was developed to evaluate the sensitivity of these metrics to dynamic attentional demands in a more naturalistic multitasking context. By manipulating attention levels through varying secondary tasks in the NASA MATB-II task while maintaining a consistent primary reaction-time task, this study successfully demonstrated the effectiveness of the paradigm. Results: Results indicate that both the beta/alpha ratio and EI are sensitive to changes in attentional demands, with beta/alpha being more responsive to dynamic variations in attention, and EI reflecting more the overall effort required to sustain performance, especially in conditions where maintaining attention is challenging. Conclusions: The potential for predicting the attention lapses through integration of performance metrics, EEG measures, and subjective assessments was demonstrated, providing a more nuanced understanding of dynamic fluctuations of attention in multitasking scenarios, mimicking those in real-world safety-critical tasks. These findings provide a foundation for advancing methods to monitor attention fluctuations accurately and mitigate risks in critical scenarios, such as train-driving or automated vehicle operation, where maintaining a high attention level is crucial.

Robots as Mental Health Coaches: A Study of Emotional Responses to Technology-Assisted Stress Management Tasks Using Physiological Signals

The current study investigated the effectiveness of social robots in facilitating stress management interventions for university students by evaluating their physiological responses. We collected electroencephalogram (EEG) brain activity and Galvanic Skin Responses (GSRs) together with self-reported questionnaires from two groups of students who practiced a deep breathing exercise either with a social robot or a laptop. From GSR signals, we obtained the change in participants' arousal level throughout the intervention, and from the EEG signals, we extracted the change in their emotional valence using the neurometric of Frontal Alpha Asymmetry (FAA). While subjective perceptions of stress and user experience did not differ significantly between the two groups, the physiological signals revealed differences in their emotional responses as evaluated by the arousal-valence model. The Laptop group tended to show a decrease in arousal level which, in some cases, was accompanied by negative valence indicative of boredom or lack of interest. On the other hand, the Robot group displayed two patterns; some demonstrated a decrease in arousal with positive valence indicative of calmness and relaxation, and others showed an increase in arousal together with positive valence interpreted as excitement. These findings provide interesting insights into the impact of social robots as mental well-being coaches on students' emotions particularly in the presence of the novelty effect. Additionally, they provide evidence for the efficacy of physiological signals as an objective and reliable measure of user experience in HRI settings.

Biosignal-integrated robotic systems with emerging trends in visual interfaces: A systematic review

Human-machine interfaces (HMI) are currently a trendy and rapidly expanding area of research. Interestingly, the human user does not readily observe the interface between humans and machines. Instead, interactions between the machine and electrical signals from the user's body are obscured by complex control algorithms. The result is effectively a one-way street, wherein data is only transmitted from human to machine. Thus, a gap remains in the literature: how can information be effectively conveyed to the user to enable mutual understanding between humans and machines? Here, this paper reviews recent advancements in biosignal-integrated wearable robotics, with a particular emphasis on "visualization"-the presentation of relevant data, statistics, and visual feedback to the user. This review article covers various signals of interest, such as electroencephalograms and electromyograms, and explores novel sensor architectures and key materials. Recent developments in wearable robotics are examined from control and mechanical design perspectives. Additionally, we discuss current visualization methods and outline the field's future direction. While much of the HMI field focuses on biomedical and healthcare applications, such as rehabilitation of spinal cord injury and stroke patients, this paper also covers less common applications in manufacturing, defense, and other domains.

Physiological Indicators of Fluency and Engagement during Sequential and Simultaneous Modes of Human-Robot Collaboration

OCCUPATIONAL APPLICATIONSAn understanding of fluency in human-robot teaming from a physiological standpoint is still incomplete. In our experimental study involving 24 participants, we designed a scenario for shared-space human-robot collaboration (HRC) for a material sorting task. When compared to a sequential mode of interaction, the simultaneous mode resulted in significantly higher perceptions of fluency and engagement, primarily by reducing human idle time. These observations were complemented by significant changes in physiological responses, such as ECG entropy and low frequency power. These responses could predict fluency and engagement with accuracies of 90 and 97%, respectively. Notably, the perception of fluency and preferred mode of interaction were influenced by individual preferences. Hence, it is crucial to consider both physiological responses and user preferences when designing HRC systems, to ensure a positive experience with the robot teammate and to foster engagement in long-term teamwork. Furthermore, these signals can be obtained using a single robust, low-cost, and comfortable sensor.

Multimodal Assessment of Cognitive Workload Using Neural, Subjective and Behavioural Measures in Smart Factory Settings

Collaborative robots (cobots) have largely replaced conventional industrial robots in today's workplaces, particularly in manufacturing setups, due to their improved performance and intelligent design. In the framework of Industry 5.0, humans are working alongside cobots to accomplish the required level of automation. However, human-robot interaction has brought up concerns regarding human factors (HF) and ergonomics. A human worker may experience cognitive stress as a result of cobots' irresponsive nature in unpredictably occurring situations, which adversely affects productivity. Therefore, there is a necessity to measure stress to enhance a human worker's performance in a human-robot collaborative environment. In this study, factory workers' mental workload was assessed using physiological, behavioural, and subjective measures. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals were collected to acquire brain signals and track hemodynamic activity, respectively. The effect of task complexity, cobot movement speed, and cobot payload capacity on the mental stress of a human worker were observed for a task designed in the context of a smart factory. Task complexity and cobot speed proved to be more impactful. As physiological measures are unbiased and more authentic means to estimate stress, eventually they may replace the other conventional measures if they prove to correlate with the results of traditional ones. Here, regression and artificial neural networks (ANN) were utilised to determine the correlation between physiological data and subjective and behavioural measures. Regression performed better for most of the targets and the best correlation (rsq-adj = 0.654146) was achieved for predicting missed beeps, a behavioural measure, using a combination of multiple EEG and fNIRS predictors. The k-nearest neighbours (KNN) algorithm was used to evaluate the accuracy of correlation between traditional measures and physiological variables, with the highest accuracy of 77.8% achieved for missed beeps as the target. Results show that physiological measures can be more insightful and have the tendency to replace other biased parameters.

Design Path for a Social Robot for Emotional Communication for Children with Autism Spectrum Disorder (ASD)

Children with autism spectrum disorder (ASD) have deficits in social interaction and expressing and understanding emotions. Based on this, robots for children with ASD have been proposed. However, few studies have been conducted about how to design a social robot for children with ASD. Non-experimental studies have been carried out to evaluate social robots; however, the general methodology that should be used to design a social robot is not clear. This study proposes a design path for a social robot for emotional communication for children with ASD following a user-centered design approach. This design path was applied to a case study and evaluated by a group of experts in psychology, human-robot interaction, and human-computer interaction from Chile and Colombia, as well as parents of children with ASD. Our results show that following the proposed design path for a social robot to communicate emotions for children with ASD is favorable.

Towards Living Machines: current and future trends of tactile sensing, grasping, and social robotics

The development of future technologies can be highly influenced by our deeper understanding of the principles that underlie living organisms. The Living Machines conference aims at presenting (among others) the interdisciplinary work of behaving systems based on such principles. Celebrating the 10 years of the conference, we present the progress and future challenges of some of the key themes presented in the robotics workshop of the Living Machines conference. More specifically, in this perspective paper, we focus on the advances in the field of biomimetics and robotics for the creation of artificial systems that can robustly interact with their environment, ranging from tactile sensing, grasping, and manipulation to the creation of psychologically plausible agents.

Perceived safety in human-cobot interaction for fixed-path and real-time motion planning algorithms

This study investigates how different motion planning algorithms, implemented on a collaborative robot (cobot), are perceived by 48 human subjects. The four implemented algorithms ensure human safety based on the concept of speed and separation monitoring, but differ based on the following characteristics: (a) the cobot motion happens either along a fixed path or with a trajectory that is continuously planned in real time via nonlinear model predictive control, to increase cobot productivity; (b) the cobot speed is further reduced-or not-in real time based on heart rate measurements, to increase perceived safety. We conclude that (1) using a fixed path-compared to real-time motion planning-may reduce productivity and, at least when heart rate measurements are not used to modify the cobot speed, increases perceived safety; (2) reducing cobot speed based on heart rate measurements reduces productivity but does not improve perceived safety; (3) perceived safety is positively affected by habituation during the experiment, and unaffected by previous experience.

Cognitive effects of prolonged continuous human-machine interaction: The case for mental state-based adaptive interfaces

Operators of complex systems across multiple domains (e.g., aviation, automotive, and nuclear power industry) are required to perform their tasks over prolonged and continuous periods of time. Mental fatigue as well as reduced cognitive flexibility, attention, and situational awareness all result from prolonged continuous use, putting at risk the safety and efficiency of complex operations. Mental state-based adaptive systems may be a solution to this problem. These systems infer the current mental state of an operator based on a selection of metrics ranging from operator independent measures (e.g., weather and time of day), to behavioral (e.g., reaction time and lane deviation) as well as physiological markers (e.g., electroencephalography and cardiac activity). The interaction between operator and system may then be adapted in one of many ways to mitigate any detected degraded cognitive state, thereby ensuring continued safety and efficiency. Depending on the task at hand and its specific problems, possible adaptations -usually based on machine learning estimations- e.g., include modifications of information, presentation modality or stimuli salience, as well as task scheduling. Research on adaptive systems is at the interface of several domains, including neuroergonomics, human factors, and human-computer interaction in an applied and ecological context, necessitating careful consideration of each of the aforementioned aspects. This article provides an overview of some of the key questions and aspects to be considered by researchers for the design of mental state-based adaptive systems, while also promoting their application during prolonged continuous use to pave the way toward safer and more efficient human-machine interaction.

Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications

Manned-Unmanned Teaming (MUM-T) can be defined as the teaming of aerial robots (artificial agents) along with a human pilot (natural agent), in which the human agent is not an authoritative controller but rather a cooperative team player. To our knowledge, no study has yet evaluated the impact of MUM-T scenarios on operators' mental workload (MW) using a neuroergonomic approach (i.e., using physiological measures), nor provided a MW estimation through classification applied on those measures. Moreover, the impact of the non-stationarity of the physiological signal is seldom taken into account in classification pipelines, particularly regarding the validation design. Therefore this study was designed with two goals: (i) to characterize and estimate MW in a MUM-T setting based on physiological signals; (ii) to assess the impact of the validation procedure on classification accuracy. In this context, a search and rescue (S&R) scenario was developed in which 14 participants played the role of a pilot cooperating with three UAVs (Unmanned Aerial Vehicles). Missions were designed to induce high and low MW levels, which were evaluated using self-reported, behavioral and physiological measures (i.e., cerebral, cardiac, and oculomotor features). Supervised classification pipelines based on various combinations of these physiological features were benchmarked, and two validation procedures were compared (i.e., a traditional one that does not take time into account vs. an ecological one that does). The main results are: (i) a significant impact of MW on all measures, (ii) a higher intra-subject classification accuracy (75%) reached using ECG features alone or in combination with EEG and ET ones with the Adaboost, Linear Discriminant Analysis or the Support Vector Machine classifiers. However this was only true with the traditional validation. There was a significant drop in classification accuracy using the ecological one. Interestingly, inter-subject classification with ecological validation (59.8%) surpassed both intra-subject with ecological and inter-subject with traditional validation. These results highlight the need for further developments to perform MW monitoring in such operational contexts.

Scheduling and Path-Planning for Operator Oversight of Multiple Robots

There is a need for semi-autonomous systems capable of performing both automated tasks and supervised maneuvers. When dealing with multiple robots or robots with high complexity (such as humanoids), we face the issue of effectively coordinating operators across robots. We build on our previous work to present a methodology for designing trajectories and policies for robots such that a few operators can supervise multiple robots. Specifically, we: (1) Analyze the complexity of the problem, (2) Design a procedure for generating policies allowing operators to oversee many robots, (3) Present a method for designing policies and robot trajectories to allow operators to oversee multiple robots, and (4) Include both simulation and hardware experiments demonstrating our methodologies.

Controlling an Industrial Robot Using a Graphic Tablet in Offline and Online Mode

The article presents the possibility of using a graphics tablet to control an industrial robot. The paper presents elements of software development for offline and online control of a robot. The program for the graphic tablet and the operator interface was developed in C# language in Visual Studio environment, while the program controlling the industrial robot was developed in RAPID language in the RobotStudio environment. Thanks to the development of a digital twin of the real robotic workstation, tests were carried out on the correct functioning of the application in offline mode (without using the real robot). The obtained results were verified in online mode (on a real production station). The developed computer programmes have a modular structure, which makes it possible to easily adapt them to one’s needs. The application allows for changing the parameters of the robot and the parameters of the path drawing. Tests were carried out on the influence of the sampling frequency and the tool diameter on the quality of the reconstructed trajectory of the industrial robot. The results confirmed the correctness of the application. Thanks to the new method of robot programming, it is possible to quickly modify the path by the operator, without the knowledge of robot programming languages. Further research will focus on analyzing the influence of screen resolution and layout scale on the accuracy of trajectory generation.

Scheduling and Rescheduling Operations Using Decision Support Systems: Insights From Emotional Influences on Decision-Making

For many years, manufacturers have focused on improving their productivity. Production scheduling operations are critical for this objective. However, in modern manufacturing systems, the original schedule must be regularly updated as it takes places in a dynamic and uncertain environment. The modern manufacturing environment is therefore very stressful for the managers in charge of the production process because they have to cope with many disruptions and uncertainties. To help them in their decision-making process, several decision support systems (DSSs) have been developed. A recent and enormous challenge is the implementation of DSSs to efficiently manage the aforementioned issues. Nowadays, these DSSs are assumed to reduce the users' stress and workload because they automatically (re)schedule the production by applying algorithms. However, to the best of our knowledge, the reciprocal influence of users' mental state (i.e., cognitive and affective states) and the use of these DSSs have received limited attention in the literature. Particularly, the influence of users' unrelated emotions has received even less attention. However, these influences are of particular interest because they can account for explaining the efficiency of DSSs, especially in modulating DSS feedback processing. As a result, we assumed that investigating the reciprocal influences of DSSs and users' mental states could provide useful avenues of investigation. The intention of this article is then to provide recommendations for future research on scheduling and rescheduling operations by suggesting the investigation of users' mental state and encouraging to conduct such research within the neuroergonomic approach.