Improved multi-resolution foveated laparoscope with real-time digital transverse chromatic correction

A Monocular Variable Magnifications 3D Laparoscope System Using Double Liquid Lenses

During minimal invasive surgery (MIS), the laparoscope only provides a single viewpoint to the surgeon, leaving a lack of 3D perception. Many works have been proposed to obtain depth and 3D reconstruction by designing a new optical structure or by depending on the camera pose and image sequences. Most of these works modify the structure of the conventional laparoscopes and cannot provide 3D reconstruction of different magnification views. In this study, we propose a laparoscopic system based on double liquid lenses, which provide doctors with variable magnification rates, near observation, and real-time monocular 3D reconstruction. Our system composes of an optical structure that can obtain auto magnification change and autofocus without any physically moving element, and a deep learning network based on the Depth from Defocus (DFD) method, trained to suit inconsistent camera intrinsic situations and estimate depth from images of different focal lengths. The optical structure is portable and can be mounted on conventional laparoscopes. The depth estimation network estimates depth in real-time from monocular images of different focal lengths and magnification rates. Experiments show that our system provides a 0.68-1.44x zoom rate and can estimate depth from different magnification rates at 6fps. Monocular 3D reconstruction reaches at least 6mm accuracy. The system also provides a clear view even under 1mm close working distance. Ex-vivo experiments and implementation on clinical images prove that our system provides doctors with a magnified clear view of the lesion, as well as quick monocular depth perception during laparoscopy, which help surgeons get better detection and size diagnosis of the abdomen during laparoscope surgeries.

Deep Learning Enables Optofluidic Zoom System with Large Zoom Ratio and High Imaging Resolution

Due to the relatively low optical power of a liquid lens, it is usually difficult to achieve a large zoom ratio and a high-resolution image simultaneously in an optofluidic zoom imaging system. We propose an electronically controlled optofluidic zoom imaging system combined with deep learning, which achieves a large continuous zoom change and a high-resolution image. The zoom system consists of an optofluidic zoom objective and an image-processing module. The proposed zoom system can achieve a large tunable focal length range from 4.0 mm to 31.3 mm. In the focal length range of 9.4 mm to 18.8 mm, the system can dynamically correct the aberrations by six electrowetting liquid lenses to ensure the image quality. In the focal length range of 4.0-9.4 mm and 18.8-31.3 mm, the optical power of a liquid lens is mainly used to enlarge the zoom ratio, and deep learning enables the proposed zoom system with improved image quality. The zoom ratio of the system reaches 7.8×, and the maximum field of view of the system can reach ~29°. The proposed zoom system has potential applications in camera, telescope and so on.

A dual-view multi-resolution laparoscope for safer and more efficient minimally invasive surgery

Minimally invasive surgery (MIS) is limited in safety and efficiency by the hand-held nature and narrow fields of view of traditional laparoscopes. A multi-resolution foveated laparoscope (MRFL) was invented to address these concerns. The MRFL is a stationary dual-view imaging device with optical panning and zooming capabilities. It is designed to simultaneously capture and display a zoomed view and supplemental wide view of the surgical field. Optical zooming and panning capabilities facilitate repositioning of the zoomed view without physically moving the system. Additional MRFL features designed to improve safety and efficiency include its snub-nosed endoscope, tool-tip auto tracking, programmable focus profiles, unique selectable display modalities, foot pedal controls, and independently controlled surgeon and assistant displays. An MRFL prototype was constructed to demonstrate and test these features. Testing of the prototype validates its design architecture and confirms the functionality of its features. The current MRFL prototype functions adequately as a proof of concept, but the system features and performance require further improvement to be practical for clinical use.

Calibration of transverse ray and pupil aberrations for light field cameras

The accuracy of reconstructing depth maps or performing digital refocusing in light field cameras depends largely on how well the spatial and angular samples of light rays can be obtained. Ray sample errors induced by optical aberrations in a light field camera may be digitally corrected using the ray tracing data when its nominal lens design is available. However, the most commonly nominal lens prescription is not accessible to end users. Additionally, even if available, due to tolerances in optomechanical design, the ray tracing data can be inaccurate. We propose a calibration method based on measurements of fiducial markers on a checkerboard for modeling the imaging properties of light field cameras. The calibration accounts for vignetting, transverse ray errors, as well as pupil aberration, and can be applied to light field camera modeling of arbitrary pupil sampling systems. We further demonstrate the utility of the method for calibrating a tri-aperture camera that captures simultaneous stereo views via artificially induced transverse ray errors.

High-throughput multi-resolution foveated laparoscope for minimally invasive surgery

Feasibility and clinical utility of a multi-resolution foveated laparoscope (MRFL) was previously tested in a porcine surgical study. The study revealed several clinical limitations of the system including moisture proofing, working distance, image quality, low light performance, color accuracy, size, and weight. In this paper, we discuss the root causes of these limitations and strategies to correct them, present the design and prototyping of a new high throughput multi resolution foveated laparoscope (HT-MRFL), and demonstrate the HT-MRFL prototype performance in comparison to the MRFL and simulated performance metrics.

Surgeon Assessment of a Novel Multi-Resolution Foveated Laparoscope

BACKGROUND

We developed a multi-resolution foveated laparoscope (MRFL) to improve situational awareness in laparoscopic surgery. We assessed surgeon objective task performance and subjective attitudes with MRFL when used for box trainer tasks and porcine surgery.

METHODS

The MRFL simultaneously obtains a wide-angle view and a magnified view. The 2 images are displayed simultaneously. 6 urologists and 2 general surgeons performed box trainer and porcine surgery tasks with the MRFL and a standard laparoscope. Task time, use of display options, and subjective assessments were obtained.

RESULTS

Subjectively, surgeons rated situational awareness, depth perception, and instrument interference as comparable between the prototype MRFL and laparoscope for trainer tasks. For porcine surgery, the MRFL was rated as having less interference than the standard laparoscope. The image quality of the MRFL was rated interior to the standard laparoscope. Participants found the different viewing modes useful for different roles and reported that they would likely use the MRFL for conventional laparoscopic and robotic surgery. Objectively, box trainer task time was comparable for 2 of 3 tasks with the remaining task shorter for the standard laparoscope. Porcine nephrectomy and oophorectomy were feasible with the prototype MRFL, although nephrectomy task time was significantly longer than traditional laparoscopy.

CONCLUSIONS

The MRFL demonstrated feasibility for performing complex surgery. Surgeons had favorable attitudes toward its features and likelihood to use the device if available. Users utilized different view types for different tasks. Longer MRFL task times were attributed to poorer image quality of the prototype.

Prism-based tri-aperture laparoscopic objective for multi-view acquisition

This paper presents the design and prototype of a novel tri-aperture monocular laparoscopic objective that can acquire both stereoscopic views for depth information and a wide field of view (FOV) for situational awareness. The stereoscopic views are simultaneously captured via a shared objective with two displaced apertures and a custom prism. Overlapping crosstalk between the stereoscopic views is diminished by incorporating a strategically placed vignetting aperture. Meanwhile, the wide FOV is captured via a central third aperture of the same objective and provides a 2D view of the surgical field 2x as large as the area imaged by the stereoscopic views. We also demonstrate how the wide FOV provides a reference data set for stereo calibration, which enables absolute depth mapping in our experimental prototype.