A development of assistant surgical robot system based on surgical-operation-by-wire and hands-on-throttle-and-stick

Speech-mediated manipulation of da Vinci surgical system for continuous surgical flow

With the advent of robot-assisted surgery, user-friendly technologies have been applied to the da Vinci surgical system (dVSS), and their efficacy has been validated in worldwide surgical fields. However, further improvements are required to the traditional manipulation methods, which cannot control an endoscope and surgical instruments simultaneously. This study proposes a speech recognition control interface (SRCI) for controlling the endoscope via speech commands while manipulating surgical instruments to replace the traditional method. The usability-focused comparisons of the newly proposed SRCI-based and the traditional manipulation method were conducted based on ISO 9241-11. 20 surgeons and 18 novices evaluated both manipulation methods through the line tracking task (LTT) and sea spike pod task (SSPT). After the tasks, they responded to the globally reliable questionnaires: after-scenario questionnaire (ASQ), system usability scale (SUS), and NASA task load index (TLX). The completion times in the LTT and SSPT using the proposed method were 44.72% and 26.59% respectively less than the traditional method, which shows statistically significant differences (p < 0.001). The overall results of ASQ, SUS, and NASA TLX were positive for the proposed method, especially substantial reductions in the workloads such as physical demands and efforts (p < 0.05). The proposed speech-mediated method can be a candidate suitable for the simultaneous manipulation of an endoscope and surgical instruments in dVSS-used robotic surgery. Therefore, it can replace the traditional method when controlling the endoscope while manipulating the surgical instruments, which contributes to enabling the continuous surgical flow in operations consequentially.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13534-024-00429-5.

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Robot-assisted surgery platforms are utilized globally thanks to their stereoscopic vision systems and enhanced functional assistance. However, the necessity of ergonomic improvement for their use by surgeons has been increased. In surgical robots, issues with chronic fatigue exist owing to the fixed posture of the conventional stereo viewer (SV) vision system. A head-mounted display was adopted to alleviate the inconvenience, and a virtual vision platform (VVP) is proposed in this study. The VVP can provide various critical data, including medical images, vital signs, and patient records, in three-dimensional virtual reality space so that users can access medical information simultaneously. An availability of the VVP was investigated based on various user evaluations by surgeons and novices, who executed the given tasks and answered questionnaires. The performances of the SV and VVP were not significantly different; however, the craniovertebral angle of the VVP was 16.35° higher on average than that of the SV. Survey results regarding the VVP were positive; participants indicated that the optimal number of displays was six, preferring the 2 × 3 array. Reflecting the tendencies, the VVP can be a neoconceptual candidate to be customized for medical use, which opens a new prospect in a next-generation surgical robot.

Computer vision for total laparoscopic hysterectomy

INTRODUCTION

Laparoscopic surgery is widely performed in various surgical fields, but this technique requires time for surgeons to master. However, at the same time, there are many advantages in visualizing the operative field through a camera. In other words, we can visualize what we cannot see with our own eyes by using augmented reality and computer vision. Therefore, we investigated the possibilities and usefulness of computer vision in total laparoscopic hysterectomy.

METHODS

This study was approved by the Mitsui Memorial Hospital ethics committee. Patients who underwent total laparoscopic hysterectomy at Mitsui Memorial Hospital from January 2015 to December 2015 were enrolled. We evaluated 19 cases in which total laparoscopic hysterectomy was performed by the same operator and assistant. We used the Open Source Computer Vision Library for computer vision analysis. The development platform used in this study was a computer operating on Mac OS X 10.11.3.

RESULTS

We created panoramic images by matching features with the AKAZE algorithm. Noise reduction methods improved haziness caused by using energy devices. By abstracting the color of the suture string, we succeeded in abstracting the suture string from movies. We could not achieve satisfactory results in detecting ureters, and we expect that creative ideas for ureter detection may arise from collaborations between surgeons and medical engineers.

CONCLUSIONS

Although this was a preliminary study, the results suggest the utility of computer vision in assisting laparoscopic surgery.

Development of stereo endoscope system with its innovative master interface for continuous surgical operation

Background

Although robotic laparoscopic surgery has various benefits when compared with conventional open surgery and minimally invasive surgery, it also has issues to overcome and one of the issues is the discontinuous surgical flow that occurs whenever control is swapped between the endoscope system and the operating robot arm system. This can lead to problems such as collision between surgical instruments, injury to patients, and increased operation time. To achieve continuous surgical operation, a wireless controllable stereo endoscope system is proposed which enables the simultaneous control of the operating robot arm system and the endoscope system.

Methods

The proposed system consists of two improved novel master interfaces (iNMIs), a four-degrees of freedom (4-DOFs) endoscope control system (ECS), and a simple three-dimensional (3D) endoscope. In order to simultaneously control the proposed system and patient side manipulators of da Vinci research kit (dVRK), the iNMIs are installed to the master tool manipulators of dVRK system. The 4-DOFs ECS consists of four servo motors and employs a two-parallel link structure to provide translational and fulcrum point motion to the simple 3D endoscope. The images acquired by the endoscope undergo stereo calibration and rectification to provide a clear 3D vision to the surgeon as available in clinically used da Vinci surgical robot systems. Tests designed to verify the accuracy, data transfer time, and power consumption of the iNMIs were performed. The workspace was calculated to estimate clinical applicability and a modified peg transfer task was conducted with three novice volunteers.

Results

The iNMIs operated for 317 min and moved in accordance with the surgeon’s desire with a mean latency of 5 ms. The workspace was calculated to be 20378.3 cm³, which exceeds the reference workspace of 549.5 cm³. The novice volunteers were able to successfully execute the modified peg transfer task designed to evaluate the proposed system’s overall performance.

Conclusions

The experimental results verify that the proposed 3D endoscope system enables continuous surgical flow. The workspace is suitable for the performance of numerous types of surgeries. Therefore, the proposed system is expected to provide much higher safety and efficacy for current surgical robot systems.