A Systematic Review of Multilateral Teleoperation Systems

Telesurgery and the importance of context

Telesurgery has the potential to overcome geographical barriers in surgical care, encouraging its deployment in areas with sparse surgical expertise. Despite successful in-human experiments and substantial technological progress, the adoption of telesurgery remains slow. In this Review, we analyze the reasons for this slow adoption. First, we identify various contexts for telesurgery and highlight the vastly different requirements for their realization. We then discuss why procedures with high urgency and skill sparsity are particularly suitable for telesurgery. Last, we summarize key research areas essential for further progress. The goal of this Review is to provide the reader with a comprehensive analysis of the current state of telesurgery research and to provide guidance for faster adoption of this exciting technology.

Walking and scuba diving assisted amphibious exoskeleton robots: the designing of power assist control and myoelectricity based wearers' fatigue evaluation

Exoskeleton robots have the potential to augment human motor capabilities. however, current control strategies often require task-specific control laws tailored for different scenarios, which limits the applicability of exoskeletons. In this study, we propose a control strategy for exoskeleton robots that is adaptable across various scenarios. We employ adaptive oscillators (AO) with feedback control to rapidly estimate the wearer's motion phase and subsequently provide torque assistance to the wearer's hip joint based on a TCN-LSTM model. During experiments, we collected surface electromyographic (sEMG) signals from the tibialis anterior, gastrocnemius, and rectus muscles of seven groups of subjects performing treadmill walking and inclined treadmill exercises. We utilized the short-time Fourier transform to extract frequency characteristics of the signals and statistically analyzed the rate of frequency change in each muscle group under different strategies. The results indicate that when wearing the exoskeleton, the overall muscle frequency changes more slowly, suggesting that subjects can maintain activity for a longer duration before fatigue sets in. This control strategy effectively reduces the energetic cost of lower limb work for the wearer and enhances the exoskeleton's versatility in various applications.

Interday Reliability of Upper-limb Geometric MyoPassivity Map for Physical Human-Robot Interaction

The value of intrinsic energetic behavior of human biomechanics has recently been recognized and exploited in physical human-robot interaction (pHRI). The authors have recently proposed the concept of "Biomechanical Excess of Passivity," based on nonlinear control theory, to construct a user-specific energetic map. The map would assess the behavior of the upper-limb in absorbing the kinesthetic energy when interacting with robots. Integrating such knowledge into the design of pHRI stabilizers can reduce the conservatism of the control by unleashing hidden energy reservoirs indicating a less conservative margin of stability. The outcome would enhance the system's performance, such as rendering kinesthetic transparency of (tele)haptics systems. However, current methods require an offline data-driven identification procedure prior to each operation to estimate the energetic map of human biomechanics. This can be time-consuming and challenge users susceptible to fatigue. In this study, for the first time, we investigate the interday reliability of upper-limb passivity maps in a sample of five healthy subjects. Our statistical analyses indicate that the identified passivity map is highly reliable in estimating the expected energetic behavior based on Intraclass correlation coefficient analysis (conducted on different days and with various interactions). The results illustrate that a one-shot estimate is a reliable measure to be used repeatedly in biomechanics-aware pHRI stabilization, enhancing practicality in real-life scenarios.

Toward immersive communications in 6G

The sixth generation (6G) networks are expected to enable immersive communications and bridge the physical and the virtual worlds. Integrating extended reality, holography, and haptics, immersive communications will revolutionize how people work, entertain, and communicate by enabling lifelike interactions. However, the unprecedented demand for data transmission rate and the stringent requirements on latency and reliability create challenges for 6G networks to support immersive communications. In this survey article, we present the prospect of immersive communications and investigate emerging solutions to the corresponding challenges for 6G. First, we introduce use cases of immersive communications, in the fields of entertainment, education, and healthcare. Second, we present the concepts of immersive communications, including extended reality, haptic communication, and holographic communication, their basic implementation procedures, and their requirements on networks in terms of transmission rate, latency, and reliability. Third, we summarize the potential solutions to addressing the challenges from the aspects of communication, computing, and networking. Finally, we discuss future research directions and conclude this study.

Artificial Intelligence Applications and Self-Learning 6G Networks for Smart Cities Digital Ecosystems: Taxonomy, Challenges, and Future Directions

The recent upsurge of smart cities’ applications and their building blocks in terms of the Internet of Things (IoT), Artificial Intelligence (AI), federated and distributed learning, big data analytics, blockchain, and edge-cloud computing has urged the design of the upcoming 6G network generation, due to their stringent requirements in terms of the quality of services (QoS), availability, and dependability to satisfy a Service-Level-Agreement (SLA) for the end users. Industries and academia have started to design 6G networks and propose the use of AI in its protocols and operations. Published papers on the topic discuss either the requirements of applications via a top-down approach or the network requirements in terms of agility, performance, and energy saving using a down-top perspective. In contrast, this paper adopts a holistic outlook, considering the applications, the middleware, the underlying technologies, and the 6G network systems towards an intelligent and integrated computing, communication, coordination, and decision-making ecosystem. In particular, we discuss the temporal evolution of the wireless network generations’ development to capture the applications, middleware, and technological requirements that led to the development of the network generation systems from 1G to AI-enabled 6G and its employed self-learning models. We provide a taxonomy of the technology-enabled smart city applications’ systems and present insights into those systems for the realization of a trustworthy and efficient smart city ecosystem. We propose future research directions in 6G networks for smart city applications.

Simulative Evaluation of a Joint-Cartesian Hybrid Motion Mapping for Robot Hands Based on Spatial In-Hand Information

Two sub-problems are typically identified for the replication of human finger motions on artificial hands: the measurement of the motions on the human side and the mapping method of human hand movements (primary hand) on the robotic hand (target hand). In this study, we focus on the second sub-problem. During human to robot hand mapping, ensuring natural motions and predictability for the operator is a difficult task, since it requires the preservation of the Cartesian position of the fingertips and the finger shapes given by the joint values. Several approaches have been presented to deal with this problem, which is still unresolved in general. In this work, we exploit the spatial information available in-hand, in particular, related to the thumb-finger relative position, for combining joint and Cartesian mappings. In this way, it is possible to perform a large range of both volar grasps (where the preservation of finger shapes is more important) and precision grips (where the preservation of fingertip positions is more important) during primary-to-target hand mappings, even if kinematic dissimilarities are present. We therefore report on two specific realizations of this approach: a distance-based hybrid mapping, in which the transition between joint and Cartesian mapping is driven by the approaching of the fingers to the current thumb fingertip position, and a workspace-based hybrid mapping, in which the joint–Cartesian transition is defined on the areas of the workspace in which thumb and fingertips can get in contact. The general mapping approach is presented, and the two realizations are tested. In order to report the results of an evaluation of the proposed mappings for multiple robotic hand kinematic structures (both industrial grippers and anthropomorphic hands, with a variable number of fingers), a simulative evaluation was performed.

Sensors Allocation and Observer Design for Discrete Bilateral Teleoperation Systems with Multi-Rate Sampling

This study addresses sensor allocation by analyzing exponential stability for discrete-time teleoperation systems. Previous studies mostly concentrate on the continuous-time teleoperation systems and neglect the management of significant practical phenomena, such as data-swap, the effect of sampling rates of samplers, and refresh rates of actuators on the system's stability. A multi-rate sampling approach is proposed in this study, given the isolation of the master and slave robots in teleoperation systems which may have different hardware restrictions. This architecture collects data through numerous sensors with various sampling rates, assuming that a continuous-time controller stabilizes a linear teleoperation system. The aim is to assign each position and velocity signals to sensors with different sampling rates and divide the state vector between sensors to guarantee the stability of the resulting multi-rate sampled-data teleoperation system. Sufficient Krasovskii-based conditions will be provided to preserve the exponential stability of the system. This problem will be transformed into a mixed-integer program with LMIs (linear matrix inequalities). These conditions are also used to design the observers for the multi-rate teleoperation systems whose estimation errors converge exponentially to the origin. The results are validated by numerical simulations which are useful in designing sensor networks for teleoperation systems.

Human-machine-human interaction in motor control and rehabilitation: a review

BACKGROUND

Human-human (HH) interaction mediated by machines (e.g., robots or passive sensorized devices), which we call human-machine-human (HMH) interaction, has been studied with increasing interest in the last decade. The use of machines allows the implementation of different forms of audiovisual and/or physical interaction in dyadic tasks. HMH interaction between two partners can improve the dyad's ability to accomplish a joint motor task (task performance) beyond either partner's ability to perform the task solo. It can also be used to more efficiently train an individual to improve their solo task performance (individual motor learning). We review recent research on the impact of HMH interaction on task performance and individual motor learning in the context of motor control and rehabilitation, and we propose future research directions in this area.

METHODS

A systematic search was performed on the Scopus, IEEE Xplore, and PubMed databases. The search query was designed to find studies that involve HMH interaction in motor control and rehabilitation settings. Studies that do not investigate the effect of changing the interaction conditions were filtered out. Thirty-one studies met our inclusion criteria and were used in the qualitative synthesis.

RESULTS

Studies are analyzed based on their results related to the effects of interaction type (e.g., audiovisual communication and/or physical interaction), interaction mode (collaborative, cooperative, co-active, and competitive), and partner characteristics. Visuo-physical interaction generally results in better dyadic task performance than visual interaction alone. In cases where the physical interaction between humans is described by a spring, there are conflicting results as to the effect of the stiffness of the spring. In terms of partner characteristics, having a more skilled partner improves dyadic task performance more than having a less skilled partner. However, conflicting results were observed in terms of individual motor learning.

CONCLUSIONS

Although it is difficult to draw clear conclusions as to which interaction type, mode, or partner characteristic may lead to optimal task performance or individual motor learning, these results show the possibility for improved outcomes through HMH interaction. Future work that focuses on selecting the optimal personalized interaction conditions and exploring their impact on rehabilitation settings may facilitate the transition of HMH training protocols to clinical implementations.

A Novel Training and Collaboration Integrated Framework for Human-Agent Teleoperation

Human operators have the trend of increasing physical and mental workloads when performing teleoperation tasks in uncertain and dynamic environments. In addition, their performances are influenced by subjective factors, potentially leading to operational errors or task failure. Although agent-based methods offer a promising solution to the above problems, the human experience and intelligence are necessary for teleoperation scenarios. In this paper, a truncated quantile critics reinforcement learning-based integrated framework is proposed for human–agent teleoperation that encompasses training, assessment and agent-based arbitration. The proposed framework allows for an expert training agent, a bilateral training and cooperation process to realize the co-optimization of agent and human. It can provide efficient and quantifiable training feedback. Experiments have been conducted to train subjects with the developed algorithm. The performances of human–human and human–agent cooperation modes are also compared. The results have shown that subjects can complete the tasks of reaching and picking and placing with the assistance of an agent in a shorter operational time, with a higher success rate and less workload than human–human cooperation.

Applications of Haptic Technology, Virtual Reality, and Artificial Intelligence in Medical Training During the COVID-19 Pandemic

This paper examines how haptic technology, virtual reality, and artificial intelligence help to reduce the physical contact in medical training during the COVID-19 Pandemic. Notably, any mistake made by the trainees during the education process might lead to undesired complications for the patient. Therefore, training of the medical skills to the trainees have always been a challenging issue for the expert surgeons, and this is even more challenging in pandemics. The current method of surgery training needs the novice surgeons to attend some courses, watch some procedure, and conduct their initial operations under the direct supervision of an expert surgeon. Owing to the requirement of physical contact in this method of medical training, the involved people including the novice and expert surgeons confront a potential risk of infection to the virus. This survey paper reviews recent technological breakthroughs along with new areas in which assistive technologies might provide a viable solution to reduce the physical contact in the medical institutes during the COVID-19 pandemic and similar crises.

Robotics and AI for Teleoperation, Tele-Assessment, and Tele-Training for Surgery in the Era of COVID-19: Existing Challenges, and Future Vision

The unprecedented shock caused by the COVID-19 pandemic has severely influenced the delivery of regular healthcare services. Most non-urgent medical activities, including elective surgeries, have been paused to mitigate the risk of infection and to dedicate medical resources to managing the pandemic. In this regard, not only surgeries are substantially influenced, but also pre- and post-operative assessment of patients and training for surgical procedures have been significantly impacted due to the pandemic. Many countries are planning a phased reopening, which includes the resumption of some surgical procedures. However, it is not clear how the reopening safe-practice guidelines will impact the quality of healthcare delivery. This perspective article evaluates the use of robotics and AI in 1) robotics-assisted surgery, 2) tele-examination of patients for pre- and post-surgery, and 3) tele-training for surgical procedures. Surgeons interact with a large number of staff and patients on a daily basis. Thus, the risk of infection transmission between them raises concerns. In addition, pre- and post-operative assessment also raises concerns about increasing the risk of disease transmission, in particular, since many patients may have other underlying conditions, which can increase their chances of mortality due to the virus. The pandemic has also limited the time and access that trainee surgeons have for training in the OR and/or in the presence of an expert. In this article, we describe existing challenges and possible solutions and suggest future research directions that may be relevant for robotics and AI in addressing the three tasks mentioned above.

Review: How Can Intelligent Robots and Smart Mechatronic Modules Facilitate Remote Assessment, Assistance, and Rehabilitation for Isolated Adults With Neuro-Musculoskeletal Conditions?

Worldwide, at the time this article was written, there are over 127 million cases of patients with a confirmed link to COVID-19 and about 2.78 million deaths reported. With limited access to vaccine or strong antiviral treatment for the novel coronavirus, actions in terms of prevention and containment of the virus transmission rely mostly on social distancing among susceptible and high-risk populations. Aside from the direct challenges posed by the novel coronavirus pandemic, there are serious and growing secondary consequences caused by the physical distancing and isolation guidelines, among vulnerable populations. Moreover, the healthcare system's resources and capacity have been focused on addressing the COVID-19 pandemic, causing less urgent care, such as physical neurorehabilitation and assessment, to be paused, canceled, or delayed. Overall, this has left elderly adults, in particular those with neuromusculoskeletal (NMSK) conditions, without the required service support. However, in many cases, such as stroke, the available time window of recovery through rehabilitation is limited since neural plasticity decays quickly with time. Given that future waves of the outbreak are expected in the coming months worldwide, it is important to discuss the possibility of using available technologies to address this issue, as societies have a duty to protect the most vulnerable populations. In this perspective review article, we argue that intelligent robotics and wearable technologies can help with remote delivery of assessment, assistance, and rehabilitation services while physical distancing and isolation measures are in place to curtail the spread of the virus. By supporting patients and medical professionals during this pandemic, robots, and smart digital mechatronic systems can reduce the non-COVID-19 burden on healthcare systems. Digital health and cloud telehealth solutions that can complement remote delivery of assessment and physical rehabilitation services will be the subject of discussion in this article due to their potential in enabling more effective and safer NMSDK rehabilitation, assistance, and assessment service delivery. This article will hopefully lead to an interdisciplinary dialogue between the medical and engineering sectors, stake holders, and policy makers for a better delivery of care for those with NMSK conditions during a global health crisis including future pandemics.

Review of Advanced Medical Telerobots

The advent of telerobotic systems has revolutionized various aspects of the industry and human life. This technology is designed to augment human sensorimotor capabilities to extend them beyond natural competence. Classic examples are space and underwater applications when distance and access are the two major physical barriers to be combated with this technology. In modern examples, telerobotic systems have been used in several clinical applications, including teleoperated surgery and telerehabilitation. In this regard, there has been a significant amount of research and development due to the major benefits in terms of medical outcomes. Recently telerobotic systems are combined with advanced artificial intelligence modules to better share the agency with the operator and open new doors of medical automation. In this review paper, we have provided a comprehensive analysis of the literature considering various topologies of telerobotic systems in the medical domain while shedding light on different levels of autonomy for this technology, starting from direct control, going up to command-tracking autonomous telerobots. Existing challenges, including instrumentation, transparency, autonomy, stochastic communication delays, and stability, in addition to the current direction of research related to benefit in telemedicine and medical automation, and future vision of this technology, are discussed in this review paper.

PHTNet: Characterization and Deep Mining of Involuntary Pathological Hand Tremor using Recurrent Neural Network Models

The global aging phenomenon has increased the number of individuals with age-related neurological movement disorders including Parkinson's Disease (PD) and Essential Tremor (ET). Pathological Hand Tremor (PHT), which is considered among the most common motor symptoms of such disorders, can severely affect patients' independence and quality of life. To develop advanced rehabilitation and assistive technologies, accurate estimation/prediction of nonstationary PHT is critical, however, the required level of accuracy has not yet been achieved. The lack of sizable datasets and generalizable modeling techniques that can fully represent the spectrotemporal characteristics of PHT have been a critical bottleneck in attaining this goal. This paper addresses this unmet need through establishing a deep recurrent model to predict and eliminate the PHT component of hand motion. More specifically, we propose a machine learning-based, assumption-free, and real-time PHT elimination framework, the PHTNet, by incorporating deep bidirectional recurrent neural networks. The PHTNet is developed over a hand motion dataset of 81 ET and PD patients collected systematically in a movement disorders clinic over 3 years. The PHTNet is the first intelligent systems model developed on this scale for PHT elimination that maximizes the resolution of estimation and allows for prediction of future and upcoming sub-movements.

Model Mediation to Overcome Light Limitations—Toward a Secure Tactile Internet System

Enabling haptic communication as well as voice and data over the future 5G cellular system has become a demand. Tactile Internet is one of the main use cases of the 5G system that will allow the transfer of haptic communications in real time. Latency, availability, reliability, and security are the main design challenges of the tactile Internet system and haptic based bilateral teleoperation systems. An end-to-end latency of 1 ms remains the main challenge toward tactile Internet system realization, not only for the processing and coding delays but mainly for the limitations of light. In this work, we analyze the key solutions to overcome the light limitations and enable the tactile Internet over any distances with the required latency. Building a virtual model or model mediated for the remote environment at the edge cloud unit near to the end user is the main solution. By means of AI, the virtual model can predict the behavior of the remote environment and thus, the end user can interact with the virtual environment with a high system experience. This literature review covers the existing work of the model mediated bilateral teleoperated systems and discusses its availability for the tactile Internet system. Furthermore, the security issues of tactile Internet system and the effect of model mediated system on the required security level of tactile Internet applications are discussed. Finally, a structure for the tactile Internet system, with the deployment of model mediation, is suggested.