Multi-view vision system for laparoscopy surgery

A prospective study on the enhancement of surgical safety in robotic surgery: The BirdView camera system

PURPOSE

To investigate the surgical safety and benefits of using the BirdView camera system with a wide field of view in robotic surgery for rectal cancer in a prospective clinical study.

METHODS

This study included 20 consecutive patients who underwent robotic surgery at our institution between the years 2022 and 2023. The primary endpoint was perioperative safety, which was defined as the occurrence of adverse events, including other organ injuries and malfunctions, caused by the BirdView camera system.

RESULTS

There were no injuries to any other organs caused by the console surgeon or assistant forceps during surgery. Surgical adverse events occurred in five cases (atelectasis, paralytic ileus, and anastomotic leakage) during the postoperative course. There were no cases of device failure or damage to the surrounding organs, including peritoneal heat damage.

CONCLUSIONS

We believe that the BirdView system could be valuable in improving the safety of robotic surgery by enabling the observation of blind spots, thus preventing harm to other organs.

Development of panorama vision ring for thoracoscopy

PURPOSE

Minimally invasive surgery (MIS) using a thoraco- or laparoscope is becoming a more common surgical technique. In MIS, a magnified view from a thoracoscope helps surgeons conduct precise operations. However, there is a risk of the visible area becoming narrow. To confirm that the operation field is safe, the surgeon will draw the thoracoscope back to check the marginal area of the target and insert it again many times during MIS. To reduce the surgeon's load, we aim to visualize the entire thoracic cavity using a newly developed device called "panorama vision ring" (PVR).

METHOD

The PVR is used instead of a wound retractor or a trocar. It is a ring-type socket with one big hole for the thoracoscope and four small holes for tiny cameras placed around the big hole. The views from the tiny cameras are fused into one wider view that visualizes the entire thoracic cavity. A surgeon can proceed with an operation by checking what exists outside of the thoracoscopic view. Also, she/he can check whether or not bleeding has occurred from the image of the entire cavity.

RESULTS

We evaluated the view-expansion ability of the PVR by using a three-dimensional full-scale thoracic model. The experimental results showed that the entire thoracic cavity could be visible in a panoramic view generated by the PVR. We also demonstrated pulmonary lobectomy in virtual MIS using the PVR. Surgeons could perform a pulmonary lobectomy while checking the entire cavity.

CONCLUSION

We developed the PVR, which uses tiny auxiliary cameras to create a panoramic view of the entire thoracic cavity during MIS. We aim to make MIS safer for patients and more comfortable for surgeons through the development of the PVR.

A dual-view multi-resolution laparoscope for safer and more efficient minimally invasive surgery

Minimally invasive surgery (MIS) is limited in safety and efficiency by the hand-held nature and narrow fields of view of traditional laparoscopes. A multi-resolution foveated laparoscope (MRFL) was invented to address these concerns. The MRFL is a stationary dual-view imaging device with optical panning and zooming capabilities. It is designed to simultaneously capture and display a zoomed view and supplemental wide view of the surgical field. Optical zooming and panning capabilities facilitate repositioning of the zoomed view without physically moving the system. Additional MRFL features designed to improve safety and efficiency include its snub-nosed endoscope, tool-tip auto tracking, programmable focus profiles, unique selectable display modalities, foot pedal controls, and independently controlled surgeon and assistant displays. An MRFL prototype was constructed to demonstrate and test these features. Testing of the prototype validates its design architecture and confirms the functionality of its features. The current MRFL prototype functions adequately as a proof of concept, but the system features and performance require further improvement to be practical for clinical use.

Laparoscopic surgery using 8 K ultra-high-definition technology: Outcomes of a phase II study

INTRODUCTION

Currently, laparoscopic surgery generally relies on 2 K high-definition image quality. The National Cancer Center Hospital, Olympus Corporation, and NHK Engineering System Inc. recently developed a new laparoscopic system with an 8 K ultra-high-definition (UHD) camera that provides images with a high-resolution, wide color range, high frame rate, and high dynamic range. This study aimed to investigate the effectiveness and safety of a new laparoscopic system which uses an 8 K UHD camera system (8K UHD system).

METHODS

This phase II study enrolled 23 patients with colon or rectosigmoid cancer who were indicated for radical resection with laparoscopic colectomy using the 8 K UHD system. The primary endpoint was the proportion of patients with ≥30 mL of intraoperative blood loss.

RESULTS

Of the 23 patients, 22 completed laparoscopic surgery with the 8 K UHD system. One patient was converted to the 2 K high-definition laparoscopic system due to technical difficulties with the 8 K UHD system during surgery. The median amount of intraoperative blood loss was 14 mL (range, 2-71 mL), and number of patients with intraoperative blood loss ≥30 mL was four (17.4%). None of the patients had >100 mL of intraoperative blood loss. No intraoperative complications were noted, and four (17.4%) patients developed postoperative complications. Pathological complete resection was achieved in all patients, and no conversion to open surgery was required.

CONCLUSIONS

Laparoscopic surgery using the 8 K UHD system appears to be both safe and effective. However, further refinements may be necessary to improve usability.

Real-Time Expanded Field-of-View for Minimally Invasive Surgery Using Multi-Camera Visual Simultaneous Localization and Mapping

Minimally invasive surgery is widely used because of its tremendous benefits to the patient. However, there are some challenges that surgeons face in this type of surgery, the most important of which is the narrow field of view. Therefore, we propose an approach to expand the field of view for minimally invasive surgery to enhance surgeons’ experience. It combines multiple views in real-time to produce a dynamic expanded view. The proposed approach extends the monocular Oriented features from an accelerated segment test and Rotated Binary robust independent elementary features—Simultaneous Localization And Mapping (ORB-SLAM) to work with a multi-camera setup. The ORB-SLAM’s three parallel threads, namely tracking, mapping and loop closing, are performed for each camera and new threads are added to calculate the relative cameras’ pose and to construct the expanded view. A new algorithm for estimating the optimal inter-camera correspondence matrix from a set of corresponding 3D map points is presented. This optimal transformation is then used to produce the final view. The proposed approach was evaluated using both human models and in vivo data. The evaluation results of the proposed correspondence matrix estimation algorithm prove its ability to reduce the error and to produce an accurate transformation. The results also show that when other approaches fail, the proposed approach can produce an expanded view. In this work, a real-time dynamic field-of-view expansion approach that can work in all situations regardless of images’ overlap is proposed. It outperforms the previous approaches and can also work at 21 fps.

Improving vision for surgeons during laparoscopy: the Enhanced Laparoscopic Vision System (ELViS)

BACKGROUND

For many abdominal surgical interventions, laparotomy has gradually been replaced by laparoscopy, with numerous benefits for the patient in terms of post-operative recovery. However, during laparoscopy, the endoscope only provides a single viewpoint to the surgeon, leaving numerous blind spots and opening the way to peri-operative adverse events. Alternative camera systems have been proposed, but many lack the requisite resolution/robustness for use during surgery or cannot provide real-time images. Here, we present the added value of the Enhanced Laparoscopic Vision System (ELViS) which overcomes these limitations and provides a broad view of the surgical field in addition to the usual high-resolution endoscope.

METHODS

Experienced laparoscopy surgeons performed several typical procedure steps on a live pig model. The time-to-completion for surgical exercises performed by conventional endoscopy and ELViS-assisted surgery was measured. A debriefing interview following each operating session was conducted by an ergonomist, and a System Usability Scale (SUS) score was determined.

RESULTS

Proof of concept of ELVIS was achieved in an animal model with seven expert surgeons without peroperative adverse events related to the surgical device. No differences were found in time-to-completion. Mean SUS score was 74.7, classifying the usability of the ELViS as "good". During the debriefing interview, surgeons highlighted several situations where the ELViS provided a real advantage (such as during instrument insertion, exploration of the abdominal cavity or for orientation during close work) and also suggested avenues for improvement of the system.

CONCLUSIONS

This first test of the ELViS prototype on a live animal model demonstrated its usability and provided promising and useful feedback for further development.

A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

With the advent of Minimally Invasive Surgery (MIS), intra-operative imaging has become crucial for surgery and therapy guidance, allowing to partially compensate for the lack of information typical of MIS. This paper reviews the advancements in both classical (i.e. ultrasounds, X-ray, optical coherence tomography and magnetic resonance imaging) and more recent (i.e. multispectral, photoacoustic and Raman imaging) intra-operative imaging modalities. Each imaging modality was analyzed, focusing on benefits and disadvantages in terms of compatibility with the operating room, costs, acquisition time and image characteristics. Tables are included to summarize this information. New generation of hybrid surgical room and algorithms for real time/in room image processing were also investigated. Each imaging modality has its own (site- and procedure-specific) peculiarities in terms of spatial and temporal resolution, field of view and contrasted tissues. Besides the benefits that each technique offers for guidance, considerations about operators and patient risk, costs, and extra time required for surgical procedures have to be considered. The current trend is to equip surgical rooms with multimodal imaging systems, so as to integrate multiple information for real-time data extraction and computer-assisted processing. The future of surgery is to enhance surgeons eye to minimize intra- and after-surgery adverse events and provide surgeons with all possible support to objectify and optimize the care-delivery process.

Large-Field-of-View Visualization with Small Blind Spots Utilizing Tilted Micro-Camera Array for Laparoscopic Surgery

Existing laparoscopic surgery systems use a single laparoscope to visualize the surgical area with a limited field of view (FoV), necessitating maneuvering the laparoscope to search a target region. In some cases, the laparoscope needs to be moved from one surgical port to another one to detect target organs. These maneuvers would cause longer surgical time and degrade the efficiency of operation. We hypothesize that if an array of cameras can be deployed to provide a stitched video with an expanded FoV and small blind spots, the time required to perform multiple tasks at different sites can be significantly reduced. We developed a micro-camera array that can enlarge the FoV and reduce blind spots between the cameras by optimizing the angle of cameras. The video stream of this micro-camera array was designed to be processed in real-time to provide a stitched video with the expanded FoV. We mounted this micro-camera array to a Fundamentals of Laparoscopic Surgery (FLS) laparoscopic trainer box and designed an experiment to validate the hypothesis above. Surgeons, residents, and a medical student were recruited to perform a modified bean drop task, and the completion time was compared against that measured using a traditional single-camera laparoscope. It was observed that utilizing the micro-camera array, the completion time of the modified bean drop task was 203±55 s while using the laparoscope, the completion time was 245±114 s, with a p-value of 0.00097. It is also observed that the benefit of using an FoV-expanded camera array does not diminish for subjects who are more experienced. This test provides convincing evidence and validates the hypothesis that expanded FoV with small blind spots can reduce the operation time for laparoscopic surgical tasks.

A prospective study of the safety and usefulness of a new miniature wide-angle camera: the "BirdView camera system"

BACKGROUND

The performance of endoscopic surgery has quickly become widespread as a minimally invasive therapy. However, complications still occur due to technical difficulties. In the present study, we focused on the problem of blind spots, which is one of the several problems that occur during endoscopic surgery and developed "BirdView," a camera system with a wide field of view, with SHARP Corporation.

METHODS

In the present study, we conducted a clinical trial (Phase I) to confirm the safety and usefulness of the BirdView camera system. We herein report the results.

RESULTS

In this study, surgical adverse events were reported in 2 cases (problems with ileus and urination). There were no cases of device failure, damage to the surrounding organs, or mortality.

CONCLUSIONS

We evaluated the safety of the BirdView camera system. We believe that this camera system will contribute to the performance safe endoscopic surgery and the execution of robotic surgery, in which operators do not have the benefit of tactile feedback.

Large-Field-of-View Visualization Utilizing Multiple Miniaturized Cameras for Laparoscopic Surgery

The quality and the extent of intra-abdominal visualization are critical to a laparoscopic procedure. Currently, a single laparoscope is inserted into one of the laparoscopic ports to provide intra-abdominal visualization. The extent of this field of view (FoV) is rather restricted and may limit efficiency and the range of operations. Here we report a trocar-camera assembly (TCA) that promises a large FoV, and improved efficiency and range of operations. A video stitching program processes video data from multiple miniature cameras and combines these videos in real-time. This stitched video is then displayed on an operating monitor with a much larger FoV than that of a single camera. In addition, we successfully performed a standard and a modified bean drop task, without any distortion, in a simulator box by using the TCA and taking advantage of its FoV which is larger than that of the current laparoscopic cameras. We successfully demonstrated its improved efficiency and range of operations. The TCA frees up a surgical port and potentially eliminates the need of physical maneuvering of the laparoscopic camera, operated by an assistant.

An enhanced hybrid tracking-mosaicking approach for surgical view expansion

The aim of this work is to overcome the narrow surgical field of view problem in minimally invasive surgery. We achieve this by combining multiple views of the camera-retractable trocar which can obtain surgical viewpoints different from laparoscopic view. However, the accuracy and time are essential factors in this process. Therefore, we tend to improve the accuracy of a hybrid tracking-mosaicking approach which can combine several views at high speed. Two improvements are presented and analyzed here. The first improvement utilizes two sharping methodologies to enhance the image quality. This enhancement, in turn, improves the interest point extraction process and increases the number of extracted points. In the second enhancement, the tracking accuracy is improved by applying a filtering methodology to select the set of valid flow vectors only. This process reduces the tracking error which may accumulate during tracking. The experimental evaluation, shows that these improvements enhance the final mosaicking accuracy and allows us to construct a more accurate expanded view.

Review on Otological Robotic Systems: Toward Microrobot-Assisted Cholesteatoma Surgery

Otologic surgical procedures over time have become minimally invasive due to the development of medicine, microtechniques, and robotics. This trend then provides an expected reduction in the patient's recovery time and improvement in the accuracy of diagnosis and treatment. One of the most challenging difficulties that such techniques face are precise control of the instrument and supply of an ergonomic system to the surgeon. The objective of this literature review is to present requirements and guidelines for a surgical robotic system dedicated to middle ear surgery. This review is particularly focused on cholesteatoma surgery (diagnosis and surgical tools), which is one of the most frequent pathologies that urge for an enhanced treatment. This review also presents the current robotic systems that are implemented for otologic applications.

A hands-free region-of-interest selection interface for solo surgery with a wide-angle endoscope: preclinical proof of concept

BACKGROUND

A hands-free region-of-interest (ROI) selection interface is proposed for solo surgery using a wide-angle endoscope. A wide-angle endoscope provides images with a larger field of view than a conventional endoscope. With an appropriate selection interface for a ROI, surgeons can also obtain a detailed local view as if they moved a conventional endoscope in a specific position and direction.

METHODS

To manipulate the endoscope without releasing the surgical instrument in hand, a mini-camera is attached to the instrument, and the images taken by the attached camera are analyzed. When a surgeon moves the instrument, the instrument orientation is calculated by an image processing. Surgeons can select the ROI with this instrument movement after switching from 'task mode' to 'selection mode.' The accelerated KAZE algorithm is used to track the features of the camera images once the instrument is moved. Both the wide-angle and detailed local views are displayed simultaneously, and a surgeon can move the local view area by moving the mini-camera attached to the surgical instrument.

RESULTS

Local view selection for a solo surgery was performed without releasing the instrument. The accuracy of camera pose estimation was not significantly different between camera resolutions, but it was significantly different between background camera images with different numbers of features (P < 0.01). The success rate of ROI selection diminished as the number of separated regions increased. However, separated regions up to 12 with a region size of 160 × 160 pixels were selected with no failure. Surgical tasks on a phantom model and a cadaver were attempted to verify the feasibility in a clinical environment.

CONCLUSIONS

Hands-free endoscope manipulation without releasing the instruments in hand was achieved. The proposed method requires only a small, low-cost camera and an image processing. The technique enables surgeons to perform solo surgeries without a camera assistant.