Tactile Sensing for Minimally Invasive Surgery: Conventional Methods and Potential Emerging Tactile Technologies

Know your orientation: A viewpoint-aware framework for polyp segmentation

Automatic polyp segmentation in endoscopic images is critical for the early diagnosis of colorectal cancer. Despite the availability of powerful segmentation models, two challenges still impede the accuracy of polyp segmentation algorithms. Firstly, during a colonoscopy, physicians frequently adjust the orientation of the colonoscope tip to capture underlying lesions, resulting in viewpoint changes in the colonoscopy images. These variations increase the diversity of polyp visual appearance, posing a challenge for learning robust polyp features. Secondly, polyps often exhibit properties similar to the surrounding tissues, leading to indistinct polyp boundaries. To address these problems, we propose a viewpoint-aware framework named VANet for precise polyp segmentation. In VANet, polyps are emphasized as a discriminative feature and thus can be localized by class activation maps in a viewpoint classification process. With these polyp locations, we design a viewpoint-aware Transformer (VAFormer) to alleviate the erosion of attention by the surrounding tissues, thereby inducing better polyp representations. Additionally, to enhance the polyp boundary perception of the network, we develop a boundary-aware Transformer (BAFormer) to encourage self-attention towards uncertain regions. As a consequence, the combination of the two modules is capable of calibrating predictions and significantly improving polyp segmentation performance. Extensive experiments on seven public datasets across six metrics demonstrate the state-of-the-art results of our method, and VANet can handle colonoscopy images in real-world scenarios effectively. The source code is available at https://github.com/1024803482/Viewpoint-Aware-Network.

A Novel Soft and Inflatable Strain-based Tactile Sensing Balloon for Enhanced Diagnosis of Colorectal Cancer Polyps Via Colonoscopy

In this paper, with the goal of addressing the lack of tactile feedback in colorectal cancer (CRC) polyps diagnosis using a colonoscopy procedure, we propose the design and fabrication of a novel soft and inflatable strain-based tactile sensing balloon (SI-STSB). The proposed soft sensor features a unique stretchable sensing layer - that utilizes a liquid metal injected within spiral-shape microchannels of a stretchable substrate - and is integrated with a unique inflatable balloon mechanism. The proposed SI-STSB has been thoroughly characterized through different calibration experiments. Results demonstrate a phenomenal adjustable sensitivity with low hysteresis behavior under different experimental conditions for this sensor making it a great candidate for enhancing the existing diagnosis procedures.

Tissue palpation in endoscopy using EIT and soft actuators

The integration of soft robots in medical procedures has significantly improved diagnostic and therapeutic interventions, addressing safety concerns and enhancing surgeon dexterity. In conjunction with artificial intelligence, these soft robots hold the potential to expedite autonomous interventions, such as tissue palpation for cancer detection. While cameras are prevalent in surgical instruments, situations with obscured views necessitate palpation. This proof-of-concept study investigates the effectiveness of using a soft robot integrated with Electrical Impedance Tomography (EIT) capabilities for tissue palpation in simulated in vivo inspection of the large intestine. The approach involves classifying tissue samples of varying thickness into healthy and cancerous tissues using the shape changes induced on a hydraulically-driven soft continuum robot during palpation. Shape changes of the robot are mapped using EIT, providing arrays of impedance measurements. Following the fabrication of an in-plane bending soft manipulator, the preliminary tissue phantom design is detailed. The phantom, representing the descending colon wall, considers induced stiffness by surrounding tissues based on a mass-spring model. The shape changes of the manipulator, resulting from interactions with tissues of different stiffness, are measured, and EIT measurements are fed into a Long Short-Term Memory (LSTM) classifier. Train and test datasets are collected as temporal sequences of data from a single training phantom and two test phantoms, namely, A and B, possessing distinctive thickness patterns. The collected dataset from phantom B, which differs in stiffness distribution, remains unseen to the network, thus posing challenges to the classifier. The classifier and proposed method achieve an accuracy of 93 % and 88.1 % on phantom A and B, respectively. Classification results are presented through confusion matrices and heat maps, visualising the accuracy of the algorithm and corresponding classified tissues.

Optical force sensor based on plasmon modulated upconversion luminescence

We report a novel force sensor exploiting the interaction between plasmonic nanostructures and upconversion nanoparticles (UCNPs). The nanosensor is composed of a gold nanodisk and UCNPs separated by a flexible polymer layer. The gold nanodisk is designed to exhibit a plasmon resonance that selectively enhances one of the emission bands of the UCNPs while leaving the other ones largely unaffected. As the nanosensor is compressed or stretched by an external force, the polymer layer thickness changes, modulating the plasmon-UCNP coupling. The resulting changes in the luminescence intensity provides the basis for sensing. Furthermore, the nanosensor employs ratiometric sensing which makes it highly robust against any environmental variations. Our nanosensors exhibit two orders of magnitude higher responsivity than previously reported UCNP-based force sensors. They can be prepared as an on-chip sensor array or in a colloidal solution, making them suitable for a variety of applications in biology and robotics.

Innovations in Hybrid Laparoscopic Surgery: Integrating Advanced Technologies for Multidisciplinary Cases

Combining conventional laparoscopic techniques with cutting-edge technologies, such as robotics, improved imaging, and flexible equipment, hybrid laparoscopic techniques represent a revolutionary advancement in minimally invasive surgery. These methods have several benefits, such as increased accuracy, quicker healing periods, and fewer complications, which makes them especially useful in complicated multidisciplinary situations. The historical evolution, uses, benefits, and drawbacks of hybrid laparoscopic procedures are examined in this narrative review, which also covers urological, gastrointestinal, cardiothoracic, and gynecological surgery. The review focuses on how these methods promote interdisciplinary cooperation and creativity by enabling more accurate and successful surgical operations. It also discusses the equipment needs, integration difficulties, and technical difficulties that need to be resolved to reach the full potential of hybrid laparoscopic surgery. For hybrid laparoscopic procedures to become more widely used and effective in the future, there is a need for specialized training programs, interdisciplinary research collaborations, and ongoing technological advancements.

Integrated Sensors for Soft Medical Robotics

Minimally invasive procedures assisted by soft robots for surgery, diagnostics, and drug delivery have unprecedented benefits over traditional solutions from both patient and surgeon perspectives. However, the translation of such technology into commercialization remains challenging. The lack of perception abilities is one of the obstructive factors paramount for a safe, accurate and efficient robot-assisted intervention. Integrating different types of miniature sensors onto robotic end-effectors is a promising trend to compensate for the perceptual deficiencies in soft robots. For example, haptic feedback with force sensors helps surgeons to control the interaction force at the tool-tissue interface, impedance sensing of tissue electrical properties can be used for tumor detection. The last decade has witnessed significant progress in the development of multimodal sensors built on the advancement in engineering, material science and scalable micromachining technologies. This review article provides a snapshot on common types of integrated sensors for soft medical robots. It covers various sensing mechanisms, examples for practical and clinical applications, standard manufacturing processes, as well as insights on emerging engineering routes for the fabrication of novel and high-performing sensing devices.

An intelligent grasper to provide real-time force feedback to shorten the learning curve in laparoscopic training

BACKGROUND

A lack of force feedback in laparoscopic surgery often leads to a steep learning curve to the novices and traditional training system equipped with force feedback need a high educational cost. This study aimed to use a laparoscopic grasper providing force feedback in laparoscopic training which can assist in controlling of gripping forces and improve the learning processing of the novices.

METHODS

Firstly, we conducted a pre-experiment to verify the role of force feedback in gripping operations and establish the safe gripping force threshold for the tasks. Following this, we proceeded with a four-week training program. Unlike the novices without feedback (Group A2), the novices receiving feedback (Group B2) underwent training that included force feedback. Finally, we completed a follow-up period without providing force feedback to assess the training effect under different conditions. Real-time force parameters were recorded and compared.

RESULTS

In the pre-experiment, we set the gripping force threshold for the tasks based on the experienced surgeons' performance. This is reasonable as the experienced surgeons have obtained adequate skill of handling grasper. The thresholds for task 1, 2, and 3 were set as 0.731 N, 1.203 N and 0.938 N, respectively. With force feedback, the gripping force applied by the novices with feedback (Group B1) was lower than that of the novices without feedback (Group A1) (p < 0.005). During the training period, the Group B2 takes 6 trails to achieve gripping force of 0.635 N, which is lower than the threshold line, whereas the Group A2 needs 11 trails, meaning that the learning curve of Group B2 was significantly shorter than that of Group A2. Additionally, during the follow-up period, there was no significant decline in force learning, and Group B2 demonstrated better control of gripping operations. The training with force feedback received positive evaluations.

CONCLUSION

Our study shows that using a grasper providing force feedback in laparoscopic training can help to control the gripping force and shorten the learning curve. It is anticipated that the laparoscopic grasper equipped with FBG sensor is promising to provide force feedback during laparoscopic training, which ultimately shows great potential in laparoscopic surgery.

Impact of a pneumatic surgical robot with haptic feedback function on surgical manipulation

Although robotic-assisted surgery has the advantages of low patient burden and high precision without unsteady hand movements, the lack of tactile sensations may result in unexpected iatrogenic organ damage. The Saroa (Riverfield Inc., Tokyo, Japan) is a pneumatically driven robot that provides real-time haptic feedback to the surgeon. Using the Saroa robot, six examinees performed puffed rice transfer and four of them performed pig lung resection tasks with the feedback function turned on and off. The puffed rice transfer task consisted of transferring 20 grains of puffed rice from the left to the right compartment in the training box. The mean grasping forces during the puffed rice transfer task with the haptic feedback function turned off and on were 2.14 N and 0.63 N, respectively (P = 0.003). The mean grasping forces during the pig lung resection task were lower with the feedback turned on than turned off. The force that the forceps exerted on the grasping object was weaker in both tasks when the haptic feedback function was turned on, suggesting that the feedback function allows gentler handling of tissues, improving patient safety during robotic surgery.

Visualizing tactile feedback: an overview of current technologies with a focus on ultrasound elastography

Tissue elasticity remains an essential biomarker of health and is indicative of irregularities such as tumors or infection. The timely detection of such abnormalities is crucial for the prevention of disease progression and complications that arise from late-stage illnesses. However, at both the bedside and the operating table, there is a distinct lack of tactile feedback for deep-seated tissue. As surgical techniques advance toward remote or minimally invasive options to reduce infection risk and hasten healing time, surgeons lose the ability to manually palpate tissue. Furthermore, palpation of deep structures results in decreased accuracy, with the additional barrier of needing years of experience for adequate confidence of diagnoses. This review delves into the current modalities used to fulfill the clinical need of quantifying physical touch. It covers research efforts involving tactile sensing for remote or minimally invasive surgeries, as well as the potential of ultrasound elastography to further this field with non-invasive real-time imaging of the organ's biomechanical properties. Elastography monitors tissue response to acoustic or mechanical energy and reconstructs an image representative of the elastic profile in the region of interest. This intuitive visualization of tissue elasticity surpasses the tactile information provided by sensors currently used to augment or supplement manual palpation. Focusing on common ultrasound elastography modalities, we evaluate various sensing mechanisms used for measuring tactile information and describe their emerging use in clinical settings where palpation is insufficient or restricted. With the ongoing advancements in ultrasound technology, particularly the emergence of micromachined ultrasound transducers, these devices hold great potential in facilitating early detection of tissue abnormalities and providing an objective measure of patient health.

The Use of Tactile Sensors in Oral and Maxillofacial Surgery: An Overview

BACKGROUND

This overview aimed to characterize the type, development, and use of haptic technologies for maxillofacial surgical purposes. The work aim is to summarize and evaluate current advantages, drawbacks, and design choices of presented technologies for each field of application in order to address and promote future research as well as to provide a global view of the issue.

METHODS

Relevant manuscripts were searched electronically through Scopus, MEDLINE/PubMed, and Cochrane Library databases until 1 November 2022.

RESULTS

After analyzing the available literature, 31 articles regarding tactile sensors and interfaces, sensorized tools, haptic technologies, and integrated platforms in oral and maxillofacial surgery have been included. Moreover, a quality rating is provided for each article following appropriate evaluation metrics.

DISCUSSION

Many efforts have been made to overcome the technological limits of computed assistant diagnosis, surgery, and teaching. Nonetheless, a research gap is evident between dental/maxillofacial surgery and other specialties such as endovascular, laparoscopic, and microsurgery; especially for what concerns electrical and optical-based sensors for instrumented tools and sensorized tools for contact forces detection. The application of existing technologies is mainly focused on digital simulation purposes, and the integration into Computer Assisted Surgery (CAS) is far from being widely actuated. Virtual reality, increasingly adopted in various fields of surgery (e.g., sino-nasal, traumatology, implantology) showed interesting results and has the potential to revolutionize teaching and learning. A major concern regarding the actual state of the art is the absence of randomized control trials and the prevalence of case reports, retrospective cohorts, and experimental studies. Nonetheless, as the research is fast growing, we can expect to see many developments be incorporated into maxillofacial surgery practice, after adequate evaluation by the scientific community.

Approaches and outcomes of Robotic-Assisted Thoracic Surgery (RATS) for lung cancer: a narrative review

Robotic-Assisted Thoracic Surgery (RATS) is considered one of the main issues of present thoracic surgery. RATS is a minimally invasive surgical technique allowing enhanced view, accurate and complex movements, and high ergonomics for the surgeon. Despite these advantages, its application in lung procedures has been limited, mainly by its costs. Since now many different approaches have been proposed and the experience in RATS for lungs ranges from wedge resection to pneumonectomy and is mainly related to lung cancer. The present narrative review explores main approaches and outcomes of RATS lobectomy for lung cancer. A non-systematic review of literature was conducted using the PubMed search engine. An overview of lung robotic surgery is given, and main approaches of robotic lobectomy for lung cancer are exposed. Initial experiences of biportal and uniportal RATS are also described. So far, retrospective analysis reported satisfactory robotic operative outcomes, and comparison with VATS might suggest a more accurate lymphadenectomy. Some Authors might even suggest better perioperative outcomes too. From an oncological standpoint, no definitive prospective study has yet been published but several retrospective analyses report oncological outcomes comparable to those of VATS and open surgery. Literature suggests that RATS for lung procedures is safe and effective and should be considered as a valid additional surgical option.

High-Resolution Tactile-Sensation Diagnostic Imaging System for Thyroid Cancer

In this study, we propose the direct diagnosis of thyroid cancer using a small probe. The probe can easily check the abnormalities of existing thyroid tissue without relying on experts, which reduces the cost of examining thyroid tissue and enables the initial self-examination of thyroid cancer with high accuracy. A multi-layer silicon-structured probe module is used to photograph light scattered by elastic changes in thyroid tissue under pressure to obtain a tactile image of the thyroid gland. In the thyroid tissue under pressure, light scatters to the outside depending on the presence of malignant and positive properties. A simple and easy-to-use tactile-sensation imaging system is developed by documenting the characteristics of the organization of tissues by using non-invasive technology for analyzing tactile images and judging the properties of abnormal tissues.

Vibro-Acoustic Sensing of Instrument Interactions as a Potential Source of Texture-Related Information in Robotic Palpation

The direct tactile assessment of surface textures during palpation is an essential component of open surgery that is impeded in minimally invasive and robot-assisted surgery. When indirectly palpating with a surgical instrument, the structural vibrations from this interaction contain tactile information that can be extracted and analysed. This study investigates the influence of the parameters contact angle α and velocity v→ on the vibro-acoustic signals from this indirect palpation. A 7-DOF robotic arm, a standard surgical instrument, and a vibration measurement system were used to palpate three different materials with varying α and v→. The signals were processed based on continuous wavelet transformation. They showed material-specific signatures in the time-frequency domain that retained their general characteristic for varying α and v→. Energy-related and statistical features were extracted, and supervised classification was performed, where the testing data comprised only signals acquired with different palpation parameters than for training data. The classifiers support vector machine and k-nearest neighbours provided 99.67% and 96.00% accuracy for the differentiation of the materials. The results indicate the robustness of the features against variations in the palpation parameters. This is a prerequisite for an application in minimally invasive surgery but needs to be confirmed in realistic experiments with biological tissues.

Three-dimensional catheter tip force sensing using multi-core fiber Bragg gratings

Awareness of catheter tip interaction forces is a crucial aspect during cardiac ablation procedures. The most important contact forces are the ones that originate between the catheter tip and the beating cardiac tissue. Clinical studies have shown that effective ablation occurs when contact forces are in the proximity of 0.2 N. Lower contact forces lead to ineffective ablation, while higher contact forces may result in complications such as cardiac perforation. Accurate and high resolution force sensing is therefore indispensable in such critical situations. Accordingly, this work presents the development of a unique and novel catheter tip force sensor utilizing a multi-core fiber with inscribed fiber Bragg gratings. A customizable helical compression spring is designed to serve as the flexural component relaying external forces to the multi-core fiber. The limited number of components, simple construction, and compact nature of the sensor makes it an appealing solution towards clinical translation. An elaborated approach is proposed for the design and dimensioning of the necessary sensor components. The approach also presents a unique method to decouple longitudinal and lateral force measurements. A force sensor prototype and a dedicated calibration setup are developed to experimentally validate the theoretical performance. Results show that the proposed force sensor exhibits 7.4 mN longitudinal resolution, 0.8 mN lateral resolution, 0.72 mN mean longitudinal error, 0.96 mN mean lateral error, a high repeatability, and excellent decoupling between longitudinal and lateral forces.

Learning curve of robotic rectal surgery using risk-adjusted cumulative summation: a 5-year institutional experience

PURPOSE

Outline learning phases of robot-assisted laparoscopic surgery for rectal cancer and compare surgical and clinical outcomes between each phase of robot-assisted laparoscopic surgery and the mastery phase of conventional laparoscopic surgery.

METHODS

From 2015 to 2020, 210 patients underwent rectal cancer surgery at Sendai Medical Center. We performed conventional laparoscopic surgery in 110 patients and, laparoscopic surgery in 100 patients. The learning curve was evaluated using the cumulative summation method, risk-adjusted cumulative summation method, and logistic regression analysis.

RESULTS

The risk-adjusted cumulative summation learning curve was divided into three phases: phase 1 (cases 1-48), phase 2 (cases 49-80), and phase 3 (cases 81-100). Duration of hospital stay (13.1 days vs. 18.0 days, respectively; p = 0.016) and surgery (209.1 min vs. 249.5 min, respectively; p = 0.045) were significantly shorter in phase 3 of the robot-assisted laparoscopic surgery group than in the conventional laparoscopic surgery group. Blood loss volume was significantly lower in phase 1 of the robot-assisted laparoscopic surgery group than in the conventional laparoscopic surgery group (17.7 ml vs. 79.7 ml, respectively; p = 0.036). The International Prostate Symptom Score was significantly lower in the robot-assisted laparoscopic surgery group (p = 0.0131).

CONCLUSIONS

Robot-assisted laparoscopic surgery for rectal cancer was safe and demonstrated better surgical and clinical outcomes, including a shorter hospital stay, less blood loss, and a shorter surgical duration, than conventional laparoscopic surgery. After experience with at least 80 cases, tactile familiarity can be acquired from visual information only (visual haptic feedback).

CLINICAL TRIAL REGISTRATION

UMIN reference no. UMIN000019857.

A Soft Multi-Axis High Force Range Magnetic Tactile Sensor for Force Feedback in Robotic Surgical Systems

This paper presents a multi-axis low-cost soft magnetic tactile sensor with a high force range for force feedback in robotic surgical systems. The proposed sensor is designed to fully decouple the output response for normal, shear and angular forces. The proposed sensor is fabricated using rapid prototyping techniques and utilizes Neodymium magnets embedded in an elastomer over Hall sensors such that their displacement produces a voltage change that can be used to calculate the applied force. The initial spacing between the magnets and the Hall sensors is optimized to achieve a large displacement range using finite element method (FEM) simulations. The experimental characterization of the proposed sensor is performed for applied force in normal, shear and 45° angular direction. The force sensitivity of the proposed sensor in normal, shear and angular directions is 16 mV/N, 30 mV/N and 81 mV/N, respectively, with minimum mechanical crosstalk. The force range for the normal, shear and angular direction is obtained as 0–20 N, 0–3.5 N and 0–1.5 N, respectively. The proposed sensor shows a perfectly linear behavior and a low hysteresis error of 8.3%, making it suitable for tactile sensing and biomedical applications. The effect of the material properties of the elastomer on force ranges and sensitivity values of the proposed sensor is also discussed.

Stiffness Assessment and Lump Detection in Minimally Invasive Surgery Using In-House Developed Smart Laparoscopic Forceps

Minimally invasive surgery (MIS) incorporates surgical instruments through small incisions to perform procedures. Despite the potential advantages of MIS, the lack of tactile sensation and haptic feedback due to the indirect contact between the surgeon’s hands and the tissues restricts sensing the strength of applied forces or obtaining information about the biomechanical properties of tissues under operation. Accordingly, there is a crucial need for intelligent systems to provide an artificial tactile sensation to MIS surgeons and trainees. This study evaluates the potential of our proposed real-time grasping forces and deformation angles feedback to assist surgeons in detecting tissues’ stiffness. A prototype was developed using a standard laparoscopic grasper integrated with a force-sensitive resistor on one grasping jaw and a tunneling magneto-resistor on the handle’s joint to measure the grasping force and the jaws’ opening angle, respectively. The sensors’ data are analyzed using a microcontroller, and the output is displayed on a small screen and saved to a log file. This integrated system was evaluated by running multiple grasp-release tests using both elastomeric and biological tissue samples, in which the average force-to-angle-change ratio precisely resembled the stiffness of grasped samples. Another feature is the detection of hidden lumps by palpation, looking for sudden variations in the measured stiffness. In experiments, the real-time grasping feedback helped enhance the surgeons’ sorting accuracy of testing models based on their stiffness. The developed tool demonstrated a great potential for low-cost tactile sensing in MIS procedures, with room for future improvements. Significance: The proposed method can contribute to MIS by assessing stiffness, detecting hidden lumps, preventing excessive forces during operation, and reducing the learning curve for trainees.