Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance

Tactile Feedback in Robot-Assisted Minimally Invasive Surgery: A Systematic Review

BACKGROUND

Robot-assisted systems have predominantly relied on teleoperation, where visual feedback is the primary source of information. However, advances in tactile sensing and displays offer new opportunities to enhance surgical transparency, efficiency, and safety.

METHODS

A PRISMA-guided search was conducted across PubMed, IEEE Xplore, Scopus, and Web of Science databases to identify relevant studies.

RESULTS

Out of 645 screened articles, 98 met the inclusion criteria, and 33 were included in the final review. The review discusses various tactile feedback stimulus types, applications, and challenges in the context of robot-assisted minimally invasive surgery.

CONCLUSION

While kinaesthetic feedback has been extensively explored to restore the natural interaction between the surgeon and the surgical environment, tactile feedback remains largely confined to research settings. This is due to significant challenges in integrating tactile feedback into robotic systems and current limitations of sensing technologies.

Untethered Microgrippers for Biopsy in the Upper Urinary Tract

Untethered microrobots offer the possibility to perform medical interventions in anatomically complex and small regions in the body. Presently, it is necessary to access the upper urinary tract to diagnose and treat Upper Tract Urothelial Carcinoma (UTUC). Diagnostic and treatment challenges include ensuring adequate tissue sampling, accurately grading the disease, achieving completeness in endoscopic treatment, and consistently delivering medications to targeted sites. This work introduces microgrippers (µ-grippers) that are autonomously triggered by physiological temperature for biopsy in the upper urinary tract. The experiments demonstrated that µ-grippers can be deployed using standard ureteral catheters and maneuvered using an external magnetic field. The μ-grippers successfully biopsied tissue samples from ex vivo pig ureters, indicating that the thin-film bilayer springs' autonomous, physiologically triggered actuation exerts enough force to retrieve urinary tract tissue. The quality of these biopsy samples is sufficient for histopathological examination, including hematoxylin and eosin (H&E) and GATA3 immunohistochemical staining. Beyond biopsy applications, the µ-grippers' small size, wafer-scale fabrication, and multifunctionality suggest their potential for statistical sampling in the urinary tract. Experimental data and clinical reports underscore this potential through statistical simulations that compare the efficacy of µ-grippers with conventional tools, such as ureteroscopic forceps and baskets.

Investigating Haptic Feedback in Vision-Deficient Millirobot Telemanipulation

The evolution of magnetically actuated millirobots gives rise to unique teleoperation challenges due to their non-traditional kinematic and dynamic architectures, as well as their frequent use of suboptimal imaging modalities. Recent investigations into haptic interfaces for millirobots have shown promise but lack the clinically motivated task scenarios necessary to justify future development. In this work, we investigate the utility of haptic feedback on bilateral teleoperation of a magnetically actuated millirobot in visually deficient conditions. We conducted an N=23 user study in an aneurysm coiling inspired procedure, which required participants to navigate the robot through a maze in near total darkness to manipulate beads to a target under simulated fluoroscopy. We hypothesized that users will be better able to complete the telemanipulation task with haptic feedback while reducing excess forces on their surroundings compared to the no feedback conditions. Our results showed an over 40% improvement in participants' bead scoring, a nearly 10% reduction in mean force, and 13% reduction in maximum force with haptic feedback, as well as significant improvements in other metrics. Results highlight that benefits of haptic feedback are retained when haptic feedback is removed. These findings suggest that haptic feedback has the potential to significantly improve millirobot telemanipulation and control in traditionally vision deficient tasks.

Untethered Microgrippers for Precision Medicine

Microgrippers, a branch of micro/nanorobots, refer to motile miniaturized machines that are of a size in the range of several to hundreds of micrometers. Compared with tethered grippers or other microscopic diagnostic and surgical equipment, untethered microgrippers play an indispensable role in biomedical applications because of their characteristics such as miniaturized size, dexterous shape tranformation, and  controllable motion, which enables the microgrippers to enter hard-to-reach regions to execute specific medical tasks for disease diagnosis and treatment. To date, numerous medical microgrippers are developed, and their potential in cell manipulation, targeted drug delivery, biopsy, and minimally invasive surgery  are explored. To achieve controlled locomotion and efficient target-oriented actions, the materials, size, microarchitecture, and morphology of microgrippers shall be deliberately designed. In this review, the authors summarizes the latest progress in untethered micrometer-scale grippers. The working mechanisms of shape-morphing and actuation methods for effective movement are first introduced. Then, the design principle and state-of-the-art fabrication techniques of microgrippers are discussed. Finally, their applications in the precise medicine are highlighted, followed by offering future perspectives for the development of untethered medical microgrippers.

Necrobotics: Biotic Materials as Ready-to-Use Actuators

Designs perfected through evolution have informed bioinspired animal-like robots that mimic the locomotion of cheetahs and the compliance of jellyfish; biohybrid robots go a step further by incorporating living materials directly into engineered systems. Bioinspiration and biohybridization have led to new, exciting research, but humans have relied on biotic materials-non-living materials derived from living organisms-since their early ancestors wore animal hides as clothing and used bones for tools. In this work, an inanimate spider is repurposed as a ready-to-use actuator requiring only a single facile fabrication step, initiating the area of "necrobotics" in which biotic materials are used as robotic components. The unique walking mechanism of spiders-relying on hydraulic pressure rather than antagonistic muscle pairs to extend their legs-results in a necrobotic gripper that naturally resides in its closed state and can be opened by applying pressure. The necrobotic gripper is capable of grasping objects with irregular geometries and up to 130% of its own mass. Furthermore, the gripper can serve as a handheld device and innately camouflages in outdoor environments. Necrobotics can be further extended to incorporate biotic materials derived from other creatures with similar hydraulic mechanisms for locomotion and articulation.

Multi-Degree-of-Freedom Robots Powered and Controlled by Microwaves

Microwaves have become a promising wireless driving strategy due to the advantages of transmissivity through obstacles, fast energy targeting, and selective heating. Although there are some studies on microwave powered artificial muscles based on different structures, the lack of studies on microwave control has limited the development of microwave-driven (MWD) robots. Here, a far-field MWD parallel robot controlled by adjusting energy distribution via changing the polarization direction of microwaves at 2.47 GHz is first reported. The parallel robot is based on three double-layer bending actuators composed of wave-absorbing sheets and bimetallic sheets, and it can implement circular and triangular path at a distance of 0.4 m under 700 W transmitting power. The thermal response rate of the actuator under microwaves is studied, and it is found that the electric-field components can provide a faster thermal response at the optimal length of actuator than magnetic-field components. The work of the parallel robot is demonstrated in an enclosed space composed of microwave-transparent materials. This developed method demonstrates the multi-degree-of-freedom controllability for robots using microwaves and offers potential solutions for some engineering cases, such as pipeline/reactors inspection and medical applications.

Enhanced Accuracy in Magnetic Actuation: Closed-loop Control of a Magnetic Agent with Low-Error Numerical Magnetic Model Estimation

Magnetic actuation holds promise for wirelessly controlling small, magnetic surgical tools and may enable the next generation of ultra minimally invasive surgical robotic systems. Precise torque and force exertion are required for safe surgical operations and accurate state control. Dipole field estimation models perform well far from electromagnets but yield large errors near coils. Thus, manipulations near coils suffer from severe (10×) field modeling errors. We experimentally quantify closed-loop magnetic agent control performance by using both a highly erroneous dipole model and a more accurate numerical magnetic model to estimate magnetic forces and torques for any given robot pose in 2D. We compare experimental measurements with estimation errors for the dipole model and our finite element analysis (FEA) based model of fields near coils. With five different paths designed for this study, we demonstrate that FEA-based magnetic field modeling reduces positioning root-mean-square (RMS) errors by 48% to 79% as compared with dipole models. Models demonstrate close agreement for magnetic field direction estimation, showing similar accuracy for orientation control. Such improved magnetic modelling is crucial for systems requiring robust estimates of magnetic forces for positioning agents, particularly in force-sensitive environments like surgical manipulation.

Contactless Haptic Display Through Magnetic Field Control

Haptic rendering enables people to touch, perceive, and manipulate virtual objects in a virtual environment. Using six cascaded identical hollow disk electromagnets and a small permanent magnet attached to an operator's finger, this paper proposes and develops an untethered haptic interface through magnetic field control. The concentric hole inside the six cascaded electromagnets provides the workspace, where the 3D position of the permanent magnet is tracked with a Microsoft Kinect sensor. The driving currents of six cascaded electromagnets are calculated in real-time for generating the desired magnetic force. Offline data from an FEA (finite element analysis) based simulation, determines the relationship between the magnetic force, the driving currents, and the position of the permanent magnet. A set of experiments including the virtual object recognition experiment, the virtual surface identification experiment, and the user perception evaluation experiment were conducted to demonstrate the proposed system, where Microsoft HoloLens holographic glasses are used for visual rendering. The proposed magnetic haptic display leads to an untethered and non-contact interface for natural haptic rendering applications, which overcomes the constraints of mechanical linkages in tool-based traditional haptic devices.

Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks

Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.

Wirelessly Actuated Thermo- and Magneto-Responsive Soft Bimorph Materials with Programmable Shape-Morphing

Soft materials that respond to wireless external stimuli are referred to as "smart" materials due to their promising potential in real-world actuation and sensing applications in robotics, microfluidics, and bioengineering. Recent years have witnessed a burst of these stimuli-responsive materials and their preliminary applications. However, their further advancement demands more versatility, configurability, and adaptability to deliver their promised benefits. Here, a dual-stimuli-responsive soft bimorph material with three configurations that enable complex programmable 3D shape-morphing is presented. The material consists of liquid crystal elastomers (LCEs) and magnetic-responsive elastomers (MREs) via facile fabrication that orthogonally integrates their respective stimuli-responsiveness without detrimentally altering their properties. The material offers an unprecedented wide design space and abundant degree-of-freedoms (DoFs) due to the LCE's programmable director field, the MRE's programmable magnetization profile, and diverse geometric configurations. It responds to wireless stimuli of the controlled magnetic field and environmental temperature. Its dual-responsiveness allows the independent control of different DoFs for complex shape-morphing behaviors with anisotropic material properties. A diverse set of in situ reconfigurable shape-morphing and an environment-aware untethered miniature 12-legged robot capable of locomotion and self-gripping are demonstrated. Such material can provide solutions for the development of future soft robotic and other functional devices.

Magnetic Resonance Guided Navigation of Untethered Microgrippers

Microsurgical tools offer a path to less invasive clinical procedures with improved access, reduced trauma, and better recovery outcomes. There are a variety of rigid and flexible endoscopic devices that have significantly advanced diagnostics and microsurgery. However, they rely on wires or tethers for guidance and operation of small end-effector tools. While untethered physiologically responsive microgrippers have been previously shown to excise tissue from deep gastrointestinal locations in animal models, there are challenges associated with guiding them along paths and moving them to specific locations. In this communication, the magnetic dipole moment of untethered thermally responsive grippers is optimized for efficient coupling to external magnetic resonance (MR) fields. Gripper encapsulation in a millimeter sized wax pellet reduces the friction with the surrounding tissue and MR Navigation (MRN) of a 700 µm sized microgripper is realized within narrow channels in tissue phantoms and in an ex vivo porcine esophagus. The results show convincing proof-of-concept evidence that it is possible to sequentially image, move, and guide a submillimeter functional microsurgical tool in tissue conduits using a commercial preclinical MR system, and when combined with prior demonstrations of physiologically responsive in vivo biopsy are an important step towards the clinical translation of untethered microtools.

External Power-Driven Microrobotic Swarm: From Fundamental Understanding to Imaging-Guided Delivery

Untethered micro/nanorobots have been widely investigated owing to their potential in performing various tasks in different environments. The significant progress in this emerging interdisciplinary field has benefited from the distinctive features of those tiny active agents, such as wireless actuation, navigation under feedback control, and targeted delivery of small-scale objects. In recent studies, collective behaviors of these tiny machines have received tremendous attention because swarming agents can enhance the delivery capability and adaptability in complex environments and the contrast of medical imaging, thus benefiting the imaging-guided navigation and delivery. In this review, we summarize the recent research efforts on investigating collective behaviors of external power-driven micro/nanorobots, including the fundamental understanding of swarm formation, navigation, and pattern transformation. The fundamental understanding of swarming tiny machines provides the foundation for targeted delivery. We also summarize the swarm localization using different imaging techniques, including the imaging-guided delivery in biological environments. By highlighting the critical steps from understanding the fundamental interactions during swarm control to swarm localization and imaging-guided delivery applications, we envision that the microrobotic swarm provides a promising tool for delivering agents in an active, controlled manner.

A Review of Magnetic Elastomers and Their Role in Soft Robotics

Soft robotics as a field of study incorporates different mechanisms, control schemes, as well as multifunctional materials to realize robots able to perform tasks inaccessible to traditional rigid robots. Conventional methods for controlling soft robots include pneumatic or hydraulic pressure sources, and some more recent methods involve temperature and voltage control to enact shape change. Magnetism was more recently introduced as a building block for soft robotic design and control, with recent publications incorporating magnetorheological fluids and magnetic particles in elastomers, to realize some of the same objectives present in more traditional soft robotics research. This review attempts to organize and emphasize the existing work with magnetism and soft robotics, specifically studies on magnetic elastomers, while highlighting potential avenues for further research enabled by these advances.

Micro/nanoscale magnetic robots for biomedical applications

Magnetic small-scale robots are devices of great potential for the biomedical field because of the several benefits of this method of actuation. Recent work on the development of these devices has seen tremendous innovation and refinement toward ​improved performance for potential clinical applications. This review briefly details recent advancements in small-scale robots used for biomedical applications, covering their design, fabrication, applications, and demonstration of ability, and identifies the gap in studies and the difficulties that have persisted in the optimization of the use of these devices. In addition, alternative biomedical applications are also suggested for some of the technologies that show potential for other functions. This study concludes that although the field of small-scale robot research is highly innovative ​there is need for more concerted efforts to improve functionality and reliability of these devices particularly in clinical applications. Finally, further suggestions are made toward ​the achievement of commercialization for these devices.

A Haptic Shared-Control Architecture for Guided Multi-Target Robotic Grasping

Although robotic telemanipulation has always been a key technology for the nuclear industry, little advancement has been seen over the last decades. Despite complex remote handling requirements, simple mechanically linked master-slave manipulators still dominate the field. Nonetheless, there is a pressing need for more effective robotic solutions able to significantly speed up the decommissioning of legacy radioactive waste. This paper describes a novel haptic shared-control approach for assisting a human operator in the sort and segregation of different objects in a cluttered and unknown environment. A three-dimensional scan of the scene is used to generate a set of potential grasp candidates on the objects at hand. These grasp candidates are then used to generate guiding haptic cues, which assist the operator in approaching and grasping the objects. The haptic feedback is designed to be smooth and continuous as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. To validate our approach, we carried out two human-subject studies, enrolling 15 participants. We registered an average improvement of 20.8%, 20.1%, and 32.5% in terms of completion time, linear trajectory, and perceived effectiveness, respectively, between the proposed approach and standard teleoperation.

Caring About the Human Operator: Haptic Shared Control for Enhanced User Comfort in Robotic Telemanipulation

Haptic shared control enables a human operator and an autonomous controller to share the control of a robotic system using haptic active constraints. It has been used in robotic teleoperation for different purposes, such as navigating along paths minimizing the torques requested to the manipulator or avoiding possibly dangerous areas of the workspace. However, few works have focused on using these ideas to account for the user's comfort. In this article, we present an innovative haptic-enabled shared control approach aimed at minimizing the user's workload during a teleoperated manipulation task. Using an inverse kinematic model of the human arm and the rapid upper limb assessment (RULA) metric, the proposed approach estimates the current user's comfort online. From this measure and an a priori knowledge of the task, we then generate dynamic active constraints guiding the users towards a successful completion of the task, along directions that improve their posture and increase their comfort. Studies with human subjects show the effectiveness of the proposed approach, yielding a 30% perceived reduction of the workload with respect to using standard guided human-in-the-loop teleoperation.

Power Wheelchair Navigation Assistance Using Wearable Vibrotactile Haptics

People with severe disabilities often rely on power wheelchairs for moving around. However, if their driving abilities are affected by their condition, driving a power wheelchair can become very dangerous, both for themselves and the surrounding environment. This article proposes the use of wearable vibrotactile haptics for wheelchair navigation assistance. We use one or two haptic armbands, each composed of four evenly-spaced vibrotactile actuators, for providing different navigation information to power wheelchair users. With respect to other available solutions, our approach provides rich navigation information while always leaving the patient in control of the wheelchair motion. Moreover, our armbands can be easily adapted for different limbs and can be used by all those patients who are unable to safely maneuver a kinesthetic interface. The results of two human subjects studies show the viability and effectiveness of the proposed technique with respect to not providing any environmental cue. Collisions were reduced by 49% when using the vibrotactile armbands. Moreover, most subjects expressed a preference for receiving haptic feedback and found the armbands comfortable to wear and use.

Soft Robotics in Minimally Invasive Surgery

Soft robotic devices have desirable traits for applications in minimally invasive surgery (MIS), but many interdisciplinary challenges remain unsolved. To understand current technologies, we carried out a keyword search using the Web of Science and Scopus databases, applied inclusion and exclusion criteria, and compared several characteristics of the soft robotic devices for MIS in the resulting articles. There was low diversity in the device designs and a wide-ranging level of detail regarding their capabilities. We propose a standardized comparison methodology to characterize soft robotics for various MIS applications, which will aid designers producing the next generation of devices.

Soft Robotic Grippers

Advances in soft robotics, materials science, and stretchable electronics have enabled rapid progress in soft grippers. Here, a critical overview of soft robotic grippers is presented, covering different material sets, physical principles, and device architectures. Soft gripping can be categorized into three technologies, enabling grasping by: a) actuation, b) controlled stiffness, and c) controlled adhesion. A comprehensive review of each type is presented. Compared to rigid grippers, end-effectors fabricated from flexible and soft components can often grasp or manipulate a larger variety of objects. Such grippers are an example of morphological computation, where control complexity is greatly reduced by material softness and mechanical compliance. Advanced materials and soft components, in particular silicone elastomers, shape memory materials, and active polymers and gels, are increasingly investigated for the design of lighter, simpler, and more universal grippers, using the inherent functionality of the materials. Embedding stretchable distributed sensors in or on soft grippers greatly enhances the ways in which the grippers interact with objects. Challenges for soft grippers include miniaturization, robustness, speed, integration of sensing, and control. Improved materials, processing methods, and sensing play an important role in future research.

A Three Revolute-Revolute-Spherical Wearable Fingertip Cutaneous Device for Stiffness Rendering

We present a novel three Revolute-Revolute-Spherical (3RRS) wearable fingertip device for the rendering of stiffness information. It is composed of a static upper body and a mobile end-effector. The upper body is located on the nail side of the finger, supporting three small servo motors, and the mobile end-effector is in contact with the finger pulp. The two parts are connected by three articulated legs, actuated by the motors. The end-effector can move toward the user's fingertip and rotate it to simulate contacts with arbitrarily-oriented surfaces. Moreover, a vibrotactile motor placed below the end-effector conveys vibrations to the fingertip. The proposed device weights 25 g for 35 x 50 x 48 mm dimensions. To test the effectiveness of our wearable haptic device and its level of wearability, we carried out two experiments, enrolling 30 human subjects in total. The first experiment tested the capability of our device in differentiating stiffness information, while the second one focused on evaluating its applicability in an immersive virtual reality scenario. Results showed the effectiveness of the proposed wearable solution, with a JND for stiffness of 208.5   17.2 N/m. Moreover, all subjects preferred the virtual interaction experience when provided with wearable cutaneous feedback, even if results also showed that subjects found our device still a bit difficult to use.

Wearable Haptic Systems for the Fingertip and the Hand: Taxonomy, Review, and Perspectives

In the last decade, we have witnessed a drastic change in the form factor of audio and vision technologies, from heavy and grounded machines to lightweight devices that naturally fit our bodies. However, only recently, haptic systems have started to be designed with wearability in mind. The wearability of haptic systems enables novel forms of communication, cooperation, and integration between humans and machines. Wearable haptic interfaces are capable of communicating with the human wearers during their interaction with the environment they share, in a natural and yet private way. This paper presents a taxonomy and review of wearable haptic systems for the fingertip and the hand, focusing on those systems directly addressing wearability challenges. The paper also discusses the main technological and design challenges for the development of wearable haptic interfaces, and it reports on the future perspectives of the field. Finally, the paper includes two tables summarizing the characteristics and features of the most representative wearable haptic systems for the fingertip and the hand.

Stimuli-Responsive Soft Untethered Grippers for Drug Delivery and Robotic Surgery

Untethered microtools that can be precisely navigated into deep in vivo locations are important for clinical procedures pertinent to minimally invasive surgery and targeted drug delivery. In this mini-review, untethered soft grippers are discussed, with an emphasis on a class of autonomous stimuli-responsive gripping soft tools that can be used to excise tissues and release drugs in a controlled manner. The grippers are composed of polymers and hydrogels and are thus compliant to soft tissues. They can be navigated using magnetic fields and controlled by robotic path-planning strategies to carry out tasks like pick-and-place of microspheres and biological materials either with user assistance, or in a fully autonomous manner. It is envisioned that the use of these untethered soft grippers will translate from laboratory experiments to clinical scenarios and the challenges that need to be overcome to make this transition are discussed.

Magnetic Motion Control and Planning of Untethered Soft Grippers using Ultrasound Image Feedback

Soft miniaturized untethered grippers can be used to manipulate and transport biological material in unstructured and tortuous environments. Previous studies on control of soft miniaturized grippers employed cameras and optical images as a feedback modality. However, the use of cameras might be unsuitable for localizing miniaturized agents that navigate within the human body. In this paper, we demonstrate the wireless magnetic motion control and planning of soft untethered grippers using feedback extracted from B-mode ultrasound images. Results show that our system employing ultrasound images can be used to control the miniaturized grippers with an average tracking error of 0.4±0.13 mm without payload and 0.36±0.05 mm when the agent performs a transportation task with a payload. The proposed ultrasound feedback magnetic control system demonstrates the ability to control miniaturized grippers in situations where visual feedback cannot be provided via cameras.