An Adaptive Shared Control Frame and Feedback Rendering in Interactive Robot-Assisted Surgical Manipulation

BACKGROUND

In an unstructured environment where real-time human decision is essential, shared control allows collaboration between humans and robotic systems, combining advantages of both. However, existing control methods are challenged with precision loss, inconsistency and interference from unconscious human inputs.

METHODS

An adaptive anisotropic control frame is presented, enabling interaction both operational and tactical levels. Using predefined trajectory, a dynamic weight function is proposed to allow the human operator to override. Movement along preferred direction is encouraged and compensated, providing accurate real-time tracking performance. Haptic feedback during shared control is evaluated and optimised.

RESULTS

Experiments validate that the raised method can achieve a tracking precision of± 0.17 m m $\pm 0.17mm$ under milling payload, with sensible feedback to the operator. The override manipulation can be rapidly made within 0.4 s as the tactical level interaction.

CONCLUSION

The proposed approach provides both stability and flexibility in interactive surgical manipulations, maintaining similar precision with autonomous execution.

SoftBoMI: a non-invasive wearable body-machine interface for mapping movement of shoulder to commands

Objective.Customized human-machine interfaces for controlling assistive devices are vital in improving the self-help ability of upper limb amputees and tetraplegic patients. Given that most of them possess residual shoulder mobility, using it to generate commands to operate assistive devices can serve as a complementary approach to brain-computer interfaces.Approach.We propose a hybrid body-machine interface prototype that integrates soft sensors and an inertial measurement unit. This study introduces both a rule-based data decoding method and a user intent inference-based decoding method to map human shoulder movements into continuous commands. Additionally, by incorporating prior knowledge of the user's operational performance into a shared autonomy framework, we implement an adaptive switching command mapping approach. This approach enables seamless transitions between the two decoding methods, enhancing their adaptability across different tasks.Main results.The proposed method has been validated on individuals with cervical spinal cord injury, bilateral arm amputation, and healthy subjects through a series of center-out target reaching tasks and a virtual powered wheelchair driving task. The experimental results show that using both the soft sensors and the gyroscope exhibits the most well-rounded performance in intent inference. Additionally, the rule-based method demonstrates better dynamic performance for wheelchair operation, while the intent inference method is more accurate but has higher latency. Adaptive switching decoding methods offer the best adaptability by seamlessly transitioning between decoding methods for different tasks. Furthermore, we discussed the differences and characteristics among the various types of participants in the experiment.Significance.The proposed method has the potential to be integrated into clothing, enabling non-invasive interaction with assistive devices in daily life, and could serve as a tool for rehabilitation assessment in the future.

User Evaluation of a Shared Robot Control System Combining BCI and Eye Tracking in a Portable Augmented Reality User Interface

This study evaluates an innovative control approach to assistive robotics by integrating brain-computer interface (BCI) technology and eye tracking into a shared control system for a mobile augmented reality user interface. Aimed at enhancing the autonomy of individuals with physical disabilities, particularly those with impaired motor function due to conditions such as stroke, the system utilizes BCI to interpret user intentions from electroencephalography signals and eye tracking to identify the object of focus, thus refining control commands. This integration seeks to create a more intuitive and responsive assistive robot control strategy. The real-world usability was evaluated, demonstrating significant potential to improve autonomy for individuals with severe motor impairments. The control system was compared with an eye-tracking-based alternative to identify areas needing improvement. Although BCI achieved an acceptable success rate of 0.83 in the final phase, eye tracking was more effective with a perfect success rate and consistently lower completion times (p<0.001). The user experience responses favored eye tracking in 11 out of 26 questions, with no significant differences in the remaining questions, and subjective fatigue was higher with BCI use (p=0.04). While BCI performance lagged behind eye tracking, the user evaluation supports the validity of our control strategy, showing that it could be deployed in real-world conditions and suggesting a pathway for further advancements.

Evaluating the real-world usability of BCI control systems with augmented reality: a user study protocol

Brain-computer interfaces (BCI) enable users to control devices through their brain activity. Motor imagery (MI), the neural activity resulting from an individual imagining performing a movement, is a common control paradigm. This study introduces a user-centric evaluation protocol for assessing the performance and user experience of an MI-based BCI control system utilizing augmented reality. Augmented reality is employed to enhance user interaction by displaying environment-aware actions, and guiding users on the necessary imagined movements for specific device commands. One of the major gaps in existing research is the lack of comprehensive evaluation methodologies, particularly in real-world conditions. To address this gap, our protocol combines quantitative and qualitative assessments across three phases. In the initial phase, the BCI prototype's technical robustness is validated. Subsequently, the second phase involves a performance assessment of the control system. The third phase introduces a comparative analysis between the prototype and an alternative approach, incorporating detailed user experience evaluations through questionnaires and comparisons with non-BCI control methods. Participants engage in various tasks, such as object sorting, picking and placing, and playing a board game using the BCI control system. The evaluation procedure is designed for versatility, intending applicability beyond the specific use case presented. Its adaptability enables easy customization to meet the specific user requirements of the investigated BCI control application. This user-centric evaluation protocol offers a comprehensive framework for iterative improvements to the BCI prototype, ensuring technical validation, performance assessment, and user experience evaluation in a systematic and user-focused manner.

Updating design guidelines for cognitive ergonomics in human-centred collaborative robotics applications: An expert survey

Within the framework of Industry 5.0, human factors are essential for enhancing the work conditions and well-being of operators interacting with even more advanced and smart manufacturing systems and machines and increasing production performances. Nevertheless, cognitive ergonomics is often underestimated when implementing advanced industrial human-robot interaction. Thus, this work aims to systematically update, develop, and validate guidelines to assist non-experts in the early stages of the design of anthropocentric and collaborative assembly applications by focusing on the main features that have positively influenced workers' cognitive responses. A methodology for structured development has been proposed. The draft guidelines have been created starting from the outcomes of a systematic and extended screening of the scientific literature. Preliminary validation has been carried out with the help of researchers working in the field. Inputs on comprehensibility and relevance have been gathered to enhance the guidelines. Lastly, a survey was used to examine in depth how international experts in different branches can interpret such guidelines. In total, 108 responders were asked to qualitatively and quantitatively evaluate the guideline's comprehensibility and provide general comments or suggestions for each guideline. Based on the survey's results, the guidelines have been validated and some have been reviewed and re-written in their final form. The present work highlights that integrating human factors into the design of collaborative applications can significantly bolster manufacturing operations' resilience through inclusivity and system adaptability by enhancing worker safety, ergonomics, and wellbeing.

Exploring Control Authority Preferences in Robotic Arm Assistance for Power Wheelchair Users

This paper uses mixed methods to explore the preliminary design of control authority preferences for an Assistive Robotic Manipulator (ARM). To familiarize users with an intelligent robotic arm, we perform two kitchen task iterations: one with user-initiated software autonomy (predefined autonomous actions) and one with manual control. Then, we introduce a third scenario, enabling users to choose between manual control and system delegation throughout the task. Results showed that, while manually switching modes and controlling the arm via joystick had a higher mental workload, participants still preferred full joystick control. Thematic analysis indicates manual control offered greater freedom and sense of accomplishment. Participants reacted positively to the idea of an interactive assistive system. Users did not want to ask the system to only assist, by taking over for certain actions, but also asked for situational feedback (e.g., 'How close am I (the gripper)?', 'Is the lid centered over the jug?'). This speaks to a future assistive system that ensures the user feels like they drive the system for the entirety of the task and provides action collaboration in addition to more granular situational awareness feedback.

Dynamic Virtual Fixture Generation Based on Intra-Operative 3D Image Feedback in Robot-Assisted Minimally Invasive Thoracic Surgery

This paper proposes a method for generating dynamic virtual fixtures with real-time 3D image feedback to facilitate human-robot collaboration in medical robotics. Seamless shared control in a dynamic environment, like that of a surgical field, remains challenging despite extensive research on collaborative control and planning. To address this problem, our method dynamically creates virtual fixtures to guide the manipulation of a trocar-placing robot arm using the force field generated by point cloud data from an RGB-D camera. Additionally, the "view scope" concept selectively determines the region for computational points, thereby reducing computational load. In a phantom experiment for robot-assisted port incision in minimally invasive thoracic surgery, our method demonstrates substantially improved accuracy for port placement, reducing error and completion time by 50% (p=1.06×10-2) and 35% (p=3.23×10-2), respectively. These results suggest that our proposed approach is promising in improving surgical human-robot collaboration.

Teleoperator-Robot-Human Interaction in Manufacturing: Perspectives from Industry, Robot Manufacturers, and Researchers

OCCUPATIONAL APPLICATIONSIndustrial robots have become an important aspect in modern industry. In the context of human-robot collaboration, enabling teleoperated robots to work in close proximity to local/onsite humans can provide new opportunities to improve human engagement in a distributed workplace. Interviews with industry stakeholders highlighted several potential benefits of such teleoperator-robot-human collaboration (tRHC), including the application of tRHC to tasks requiring both expertise and manual dexterity (e.g., maintenance and highly skilled tasks in sectors including construction, manufacturing, and healthcare), as well as opportunities to expand job accessibility for individuals with disabilities and older individuals. However, interviewees also indicated potential challenges of tRHC, particularly related to human perception (e.g., perceiving remote environments), safety, and trust. Given these challenges, and the current limited information on the practical value and implementation of tRHC, we propose several future research directions, with a focus on human factors and ergonomics, to help realize the potential benefits of tRHC.

The role of collaborative robotics in assistive and rehabilitation applications

Collaborative robotics principles and advancements may transform the field of assistive and rehabilitation robotics.

Human Factors Considerations for Quantifiable Human States in Physical Human-Robot Interaction: A Literature Review

As the global population rapidly ages with longer life expectancy and declining birth rates, the need for healthcare services and caregivers for older adults is increasing. Current research envisions addressing this shortage by introducing domestic service robots to assist with daily activities. The successful integration of robots as domestic service providers in our lives requires them to possess efficient manipulation capabilities, provide effective physical assistance, and have adaptive control frameworks that enable them to develop social understanding during human-robot interaction. In this context, human factors, especially quantifiable ones, represent a necessary component. The objective of this paper is to conduct an unbiased review encompassing the studies on human factors studied in research involving physical interactions and strong manipulation capabilities. We identified the prevalent human factors in physical human-robot interaction (pHRI), noted the factors typically addressed together, and determined the frequently utilized assessment approaches. Additionally, we gathered and categorized proposed quantification approaches based on the measurable data for each human factor. We also formed a map of the common contexts and applications addressed in pHRI for a comprehensive understanding and easier navigation of the field. We found out that most of the studies in direct pHRI (when there is direct physical contact) focus on social behaviors with belief being the most commonly addressed human factor type. Task collaboration is moderately investigated, while physical assistance is rarely studied. In contrast, indirect pHRI studies (when the physical contact is mediated via a third item) often involve industrial settings, with physical ergonomics being the most frequently investigated human factor. More research is needed on the human factors in direct and indirect physical assistance applications, including studies that combine physical social behaviors with physical assistance tasks. We also found that while the predominant approach in most studies involves the use of questionnaires as the main method of quantification, there is a recent trend that seeks to address the quantification approaches based on measurable data.

A neurorobotics approach to behaviour selection based on human activity recognition

Behaviour selection has been an active research topic for robotics, in particular in the field of human-robot interaction. For a robot to interact autonomously and effectively with humans, the coupling between techniques for human activity recognition and robot behaviour selection is of paramount importance. However, most approaches to date consist of deterministic associations between the recognised activities and the robot behaviours, neglecting the uncertainty inherent to sequential predictions in real-time applications. In this paper, we address this gap by presenting an initial neurorobotics model that embeds, in a simulated robot, computational models of parts of the mammalian brain that resembles neurophysiological aspects of the basal ganglia-thalamus-cortex (BG-T-C) circuit, coupled with human activity recognition techniques. A robotics simulation environment was developed for assessing the model, where a mobile robot accomplished tasks by using behaviour selection in accordance with the activity being performed by the inhabitant of an intelligent home. Initial results revealed that the initial neurorobotics model is advantageous, especially considering the coupling between the most accurate activity recognition approaches and the computational models of more complex animals.

Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance

Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.

FABRIC: A Framework for the Design and Evaluation of Collaborative Robots with Extended Human Adaptation

A limitation for collaborative robots (cobots) is their lack of ability to adapt to human partners, who typically exhibit an immense diversity of behaviors. We present an autonomous framework as a cobot’s real-time decision-making mechanism to anticipate a variety of human characteristics and behaviors, including human errors, toward a personalized collaboration. Our framework handles such behaviors in two levels: 1) short-term human behaviors are adapted through our novel Anticipatory Partially Observable Markov Decision Process (A-POMDP) models, covering a human’s changing intent (motivation), availability, and capability; 2) long-term changing human characteristics are adapted by our novel Adaptive Bayesian Policy Selection (ABPS) mechanism that selects a short-term decision model, e.g., an A-POMDP, according to an estimate of a human’s workplace characteristics, such as her expertise and collaboration preferences. To design and evaluate our framework over a diversity of human behaviors, we propose a pipeline where we first train and rigorously test the framework in simulation over novel human models. Then, we deploy and evaluate it on our novel physical experiment setup that induces cognitive load on humans to observe their dynamic behaviors, including their mistakes, and their changing characteristics such as their expertise. We conduct user studies and show that our framework effectively collaborates non-stop for hours and adapts to various changing human behaviors and characteristics in real-time. That increases the efficiency and naturalness of the collaboration with a higher perceived collaboration, positive teammate traits, and human trust. We believe that such an extended human-adaptation is a key to the long-term use of cobots.

Digital Twin for Human–Robot Interactions by Means of Industry 4.0 Enabling Technologies

There has been a rapid increase in the use of collaborative robots in manufacturing industries within the context of Industry 4.0 and smart factories. The existing human–robot interactions, simulations, and robot programming methods do not fit into these fast-paced technological advances as they are time-consuming, require engineering expertise, waste a lot of time in programming and the interaction is not trivial for non-expert operators. To tackle these challenges, we propose a digital twin (DT) approach for human–robot interactions (HRIs) in hybrid teams in this paper. We achieved this using Industry 4.0 enabling technologies, such as mixed reality, the Internet of Things, collaborative robots, and artificial intelligence. We present a use case scenario of the proposed method using Microsoft Hololens 2 and KUKA IIWA collaborative robot. The obtained results indicated that it is possible to achieve efficient human–robot interactions using these advanced technologies, even with operators who have not been trained in programming. The proposed method has further benefits, such as real-time simulation in natural environments and flexible system integration to incorporate new devices (e.g., robots or software capabilities).