Continuously zoom imaging probe for the multi-resolution foveated laparoscope

10× continuous optical zoom imaging using Alvarez lenses actuated by dielectric elastomers

Optical zoom is an essential function for many imaging systems including consumer electronics, biomedical microscopes, telescopes, and projectors. However, most optical zoom imaging systems have discrete zoom rates or narrow zoom ranges. In this work, a continuous optical zoom imaging system with a wide zoom range is proposed. It consists of a solid lens, two Alvarez lenses, and a camera with an objective. Each Alvarez lens is composed of two cubic phase plates, which have inverted freeform surfaces concerning each other. The movement of the cubic phase masks perpendicular to the optical axis is realized by the actuation of the dielectric elastomer. By applying actuation voltages to the dielectric elastomer, cubic phase masks are moved laterally and then the focal lengths of the two Alvarez lenses are changed. By adjusting the focal lengths of these two Alvarez lenses, the optical magnification is tuned. The proposed continuous optical zoom imaging system is built and the validity is verified by the experiments. The experimental results demonstrate that the zoom ratio is up to 10×, i.e., the magnification continuously changes from 1.58× to 15.80× when the lateral displacements of the cubic phase masks are about 1.0 mm. The rise and fall response times are 150 ms and 210 ms, respectively. The imaging resolution can reach 114 lp/mm during the optical zoom process. The proposed continuous optical imaging system is expected to be used in the fields of microscopy, biomedicine, virtual reality, etc.

Three-phase electrowetting liquid lens with deformable liquid iris

Inspired by the arrangement of iris and crystalline lens in human eyes, we propose a three-phase electrowetting liquid lens with a deformable liquid iris (TELL-DLI). The proposed electrowetting liquid lens has three-phase fluid: air, conductive liquid, and dyed insulating liquid. The insulating liquid is distributed on the inner wall of the chamber in a ring shape. By applying voltage, the contact angle is changed, so that the dyed insulating liquid contracts towards the center, which is similar to the contraction of iris and the function of crystalline lens muscle in human eyes. The variation range of focal length is from -451.9 mm to -107.9 mm. The variation range of the aperture is from 4.89 mm to 0.6 mm. Under the step voltage of 200 V, the TELL-DLI can be switched between the maximum aperture state and the zero aperture state, and the switching time is ∼150/200 ms. Because of the discrete electrodes, TELL-DLI can regionally control the shape and position of the iris, and switch between circle, ellipse, sector, and strip. The TELL-DLI has a wide application prospect in imaging systems, such as microscopic imaging system, and has the potential to be applied in the field of complex beam navigation.

Image restoration for optical zooming system based on Alvarez lenses

Alvarez lenses are known for their ability to achieve a broad range of optical power adjustment by utilizing complementary freeform surfaces. However, these lenses suffer from optical aberrations, which restrict their potential applications. To address this issue, we propose a field of view (FOV) attention image restoration model for continuous zooming. In order to simulate the degradation of optical zooming systems based on Alvarez lenses (OZA), a baseline OZA is designed where the polynomial for the Alvarez lenses consists of only three coefficients. By computing spatially varying point spread functions (PSFs), we simulate the degraded images of multiple zoom configurations and conduct restoration experiments. The results demonstrate that our approach surpasses the compared methods in the restoration of degraded images across various zoom configurations while also exhibiting strong generalization capabilities under untrained configurations.

Design of a statically foveated display based on a perceptual-driven approach

Foveated display technology has the potential to offer both a large field of view (FOV) and high spatial resolution for head-mounted display (HMD) systems, through allocating the limited resources differently between the region of interest (ROI) and the peripheral region. However, the common method used in the prior studies is based on a dual-resolution dynamic foveation scheme, which is inevitably complex and high cost due to the requirements for multiple display sources, a 2D steering mechanism, and an eye tracking device. We recently proposed a new perceptual-driven approach to design a statically foveated HMD with the goal of offering a wide FOV with nearly imperceptible or minimal degradation of the perceived image resolution within regions where frequent eye movement occurs. Compared to a dynamical dual-resolution foveation approach, it not only minimizes the hardware complexity by eliminating the need for an eyetracker, a scanning mechanism, and multiple display sources, but also offer continuous degradation in resolution to avoid visual artifacts. In this paper, a statically foveated display is designed by carefully controlling the spatial variation of optical magnification of the eyepiece optics, which covers an 80° FOV and achieves a peak resolution of 1.5 arcminutes per pixel. The angular resolution distribution of the prototype design closely matches the theoretical statically foveated scheme described in our previous work with excellent perceived performance. Finally, a foveated display prototype based on the design was experimentally demonstrated with excellent perceived performance matching the designed resolution distribution.

A dual-view multi-resolution laparoscope for safer and more efficient minimally invasive surgery

Minimally invasive surgery (MIS) is limited in safety and efficiency by the hand-held nature and narrow fields of view of traditional laparoscopes. A multi-resolution foveated laparoscope (MRFL) was invented to address these concerns. The MRFL is a stationary dual-view imaging device with optical panning and zooming capabilities. It is designed to simultaneously capture and display a zoomed view and supplemental wide view of the surgical field. Optical zooming and panning capabilities facilitate repositioning of the zoomed view without physically moving the system. Additional MRFL features designed to improve safety and efficiency include its snub-nosed endoscope, tool-tip auto tracking, programmable focus profiles, unique selectable display modalities, foot pedal controls, and independently controlled surgeon and assistant displays. An MRFL prototype was constructed to demonstrate and test these features. Testing of the prototype validates its design architecture and confirms the functionality of its features. The current MRFL prototype functions adequately as a proof of concept, but the system features and performance require further improvement to be practical for clinical use.

High-throughput multi-resolution foveated laparoscope for minimally invasive surgery

Feasibility and clinical utility of a multi-resolution foveated laparoscope (MRFL) was previously tested in a porcine surgical study. The study revealed several clinical limitations of the system including moisture proofing, working distance, image quality, low light performance, color accuracy, size, and weight. In this paper, we discuss the root causes of these limitations and strategies to correct them, present the design and prototyping of a new high throughput multi resolution foveated laparoscope (HT-MRFL), and demonstrate the HT-MRFL prototype performance in comparison to the MRFL and simulated performance metrics.

Surgeon Assessment of a Novel Multi-Resolution Foveated Laparoscope

BACKGROUND

We developed a multi-resolution foveated laparoscope (MRFL) to improve situational awareness in laparoscopic surgery. We assessed surgeon objective task performance and subjective attitudes with MRFL when used for box trainer tasks and porcine surgery.

METHODS

The MRFL simultaneously obtains a wide-angle view and a magnified view. The 2 images are displayed simultaneously. 6 urologists and 2 general surgeons performed box trainer and porcine surgery tasks with the MRFL and a standard laparoscope. Task time, use of display options, and subjective assessments were obtained.

RESULTS

Subjectively, surgeons rated situational awareness, depth perception, and instrument interference as comparable between the prototype MRFL and laparoscope for trainer tasks. For porcine surgery, the MRFL was rated as having less interference than the standard laparoscope. The image quality of the MRFL was rated interior to the standard laparoscope. Participants found the different viewing modes useful for different roles and reported that they would likely use the MRFL for conventional laparoscopic and robotic surgery. Objectively, box trainer task time was comparable for 2 of 3 tasks with the remaining task shorter for the standard laparoscope. Porcine nephrectomy and oophorectomy were feasible with the prototype MRFL, although nephrectomy task time was significantly longer than traditional laparoscopy.

CONCLUSIONS

The MRFL demonstrated feasibility for performing complex surgery. Surgeons had favorable attitudes toward its features and likelihood to use the device if available. Users utilized different view types for different tasks. Longer MRFL task times were attributed to poorer image quality of the prototype.

Prism-based tri-aperture laparoscopic objective for multi-view acquisition

This paper presents the design and prototype of a novel tri-aperture monocular laparoscopic objective that can acquire both stereoscopic views for depth information and a wide field of view (FOV) for situational awareness. The stereoscopic views are simultaneously captured via a shared objective with two displaced apertures and a custom prism. Overlapping crosstalk between the stereoscopic views is diminished by incorporating a strategically placed vignetting aperture. Meanwhile, the wide FOV is captured via a central third aperture of the same objective and provides a 2D view of the surgical field 2x as large as the area imaged by the stereoscopic views. We also demonstrate how the wide FOV provides a reference data set for stereo calibration, which enables absolute depth mapping in our experimental prototype.

Perceptual-driven approach to statically foveated head-mounted displays

A foveated display is a promising technique to realize displays offering both a large field of view (FOV) and high spatial resolution. Although several prior works have attempted to apply a foveation method to the design of a head-mounted display (HMD) system, the common method is based on a dual-resolution dynamic foveation scheme which is inevitably complex and has a high cost due to the requirements for multiple display sources, a 2D steering mechanism, and eye tracker. In this paper, a new perceptual-driven approach to the design of a statically foveated HMD is proposed with the goal of offering a wide FOV across which the degradation of the perceived image resolution is nearly imperceptible or minimal within regions of frequent eye movements. Compared to a dual-resolution discrete and dynamic foveation approach in the prior art, the static foveation approach will not only maintain resolution continuity but also eliminate the need for a scanning mechanism, multiple display sources, and an eyetracker, and therefore minimize hardware complexity. We present the general approach for creating a static foveation scheme, performance metrics for evaluating the perceived image quality, and the process of optimizing a foveation scheme to meet different requirements. Finally, we experimentally demonstrate and validate the proposed foveation scheme using a testbed system. Overall, we demonstrate a statically foveated scheme is capable of offering a display with a total 160° FOV, a constant resolution of 0.5 or 1 arcminutes per pixel within the ±10° region where frequent eye movements occur, an adequate resolution no less than 45% of peak resolution within the parafovea region of ±30°, and a data sampling efficiency as high as 90%.

High stability liquid lens with optical path modulation function

In this paper, a high stability liquid lens with optical path modulation function is designed and fabricated. The liquid lens has an outer chamber and an inner chamber, and the inner chamber has a structure with three annular anchoring layers. This structure can limit the sliding of the three-phase contact line under electrowetting effect and anchor the position of contact angle with a limited distance. The feasibility of this structure is verified by simulation and practice. The zoom imaging, contact angle, focal length and response time of the liquid lens are analyzed. The structure with three annular anchoring layers provides six anchored precision optical path modulation gears, and the optical path difference can be changed by mechanical hydraulic control, up to 1.17 mm. Widespread applications of the proposed liquid lens are foreseeable such as microscopic imaging and a telescope system, etc.

Electrically tunable lenses - eliminating mechanical axial movements during high-speed 3D live imaging

The successful investigation of photosensitive and dynamic biological events, such as those in a proliferating tissue or a dividing cell, requires non-intervening high-speed imaging techniques. Electrically tunable lenses (ETLs) are liquid lenses possessing shape-changing capabilities that enable rapid axial shifts of the focal plane, in turn achieving acquisition speeds within the millisecond regime. These human-eye-inspired liquid lenses can enable fast focusing and have been applied in a variety of cell biology studies. Here, we review the history, opportunities and challenges underpinning the use of cost-effective high-speed ETLs. Although other, more expensive solutions for three-dimensional imaging in the millisecond regime are available, ETLs continue to be a powerful, yet inexpensive, contender for live-cell microscopy.

Optical zoom imaging systems using adaptive liquid lenses

An optical zoom imaging system that can vary the magnification factor without displacing the object and the image plane has been widely used. Nonetheless, conventional optical zoom imaging systems suffer from slow response, complicated configuration, vulnerability to misalignment during zoom operation, and are incompatible with miniaturized applications. This review article focuses on state-of-the-art research on novel optical zoom imaging systems that use adaptive liquid lenses. From the aspect of the configuration, according to the number of adaptive liquid lenses, we broadly divide the current optical zoom imaging systems using adaptive liquid lenses into two configurations: multiple adaptive liquid lenses, and a single adaptive liquid lens. The principles and configurations of these optical zoom imaging systems are introduced and represented. Three different working principles of the adaptive liquid lens (liquid crystal, polymer elastic membrane, and electrowetting effect) adopted in the optical zoom imaging systems are reviewed. Some representative applications of optical zoom imaging systems using adaptive liquid lenses are introduced. The opportunities and challenges of the optical zoom imaging systems using adaptive liquid lenses are also discussed. This review aims to provide a snapshot of the current state of this research field with the aim to attract more attention to put forward the development of the next-generation optical zoom imaging systems.

Improved multi-resolution foveated laparoscope with real-time digital transverse chromatic correction

A multi-resolution foveated laparoscope (MRFL) with autofocus and zooming capabilities was previously designed to address the limiting trade-off between spatial resolution and field of view during laparoscopic minimally invasive surgery. The MRFL splits incoming light into two paths enabling simultaneous capture of the full surgical field and a zoomed-in view of the local surgical site. A fully functional prototype was constructed to demonstrate and test the autofocus, zooming capabilities, and clinical utility of this new laparoscope. The test of the prototype in both dry lab and animal models was successful, but it also revealed several major limitations of the prototype. In this paper, we present a brief overview of the aforementioned MRFL prototype design and results, and the shortcomings associated with its optical and mechanical designs. We then present several methods to address the shortcomings of the existing prototype with a modified optical layout and redesigned mechanics. The performances of the new and old system prototypes are comparatively analyzed in accordance with the design goals of the new MRFL. Finally, we present and demonstrate a real-time digital method for correcting transverse chromatic aberration to further improve the overall image quality, which can be adapted to future MRFL systems.

Speeded-Up Focus Control of Electrically Tunable Lens by Sparse Optimization

Electrically tunable lenses (ETL), also known as liquid lenses, can be focused at various distances by changing the electric signal applied on the lens. ETLs require no mechanical structures, and therefore, provide a more compact and inexpensive focus control than conventional computerized translation stages. They have been exploited in a wide range of imaging and display systems and enabled novel applications for the last several years. However, the optical fluid in the ETL is rippled after the actuation, which physically limits the response time and significantly hampers the applicability range. To alleviate this problem, we apply a sparse optimization framework that optimizes the temporal pattern of the electrical signal input to the ETL. In verification experiments, the proposed method accelerated the convergence of the focal length to the target patterns. In particular, it converged the optical power to the target at twice the speed of the simply determined input signal, and increased the quality of the captured image during multi-focal imaging.

Pre-Clinical Translation of Second Harmonic Microscopy of Meniscal and Articular Cartilage Using a Prototype Nonlinear Microendoscope

Previous studies using nonlinear microscopy have demonstrated that osteoarthritis (OA) is characterized by the gradual replacement of Type II collagen with Type I collagen. The objective of this study was to develop a prototype nonlinear laser scanning microendoscope capable of resolving the structural differences of collagen in various orthopaedically relevant cartilaginous surfaces. The current prototype developed a miniaturized femtosecond laser scanning instrument, mounted on an articulated positioning system, capable of both conventional arthroscopy and second-harmonic laser-scanning microscopy. Its optical system includes a multi-resolution optical system using a gradient index objective lens and a customized multi-purpose fiber optic sheath to maximize the collection of backscattered photons or provide joint capsule illumination. The stability and suitability of the prototype arthroscope to approach and image cartilage were evaluated through preliminary testing on fresh, minimally processed, and partially intact porcine knee joints. Image quality was sufficient to distinguish between hyaline cartilage and fibrocartilage through unique Type I and Type II collagen-specific characteristics. Imaging the meniscus revealed that the system was able to visualize differences in the collagen arrangement between the superficial and lamellar layers. Such detailed in vivo imaging of the cartilage surfaces could obviate the need to perform biopsies for ex vivo histological analysis in the future, and provide an alternative to conventional external imaging to characterize and diagnose progressive and degenerative cartilage diseases such as OA. Moreover, this system is readily customizable and may provide a suitable and modular platform for developing additional tools utilizing femtosecond lasers for tissue cutting within the familiar confines of two or three portal arthroscopy techniques.

Foveated imaging for near-eye displays

The angular resolution of current near-eye display devices is still far below human-eye acuity. How to achieve retina-level resolution while keeping wide field-of-view (FOV) remains a great challenge. In this work, we demonstrate a multi-resolution foveated display with two display panels and an optical combiner. The first display panel provides a wide FOV but relatively low resolution for the surrounding region, while the second one offers an ultra-high resolution for the central fovea region, by an optical minifying system which enhances the effective resolution by 5 ×. In addition, a switchable Pancharatnam-Berry phase deflector is employed to shift the high-resolution region. The proposed design effectively reduces the pixelation and screen-door effect in near-eye displays.

Digitally switchable multi-focal lens using freeform optics

Optical technologies offering electrically tunable optical power have found a broad range of applications, from head-mounted displays for virtual and augmented reality applications to microscopy. In this paper, we present a novel design and prototype of a digitally switchable multi-focal lens (MFL) that offers the capability of rapidly switching the optical power of the system among multiple foci. It consists of a freeform singlet and a customized programmable optical shutter array (POSA). Time-multiplexed multiple foci can be obtained by electrically controlling the POSA to switch the light path through different segments of the freeform singlet rapidly. While this method can be applied to a broad range of imaging and display systems, we experimentally demonstrate a proof-of-concept prototype for a multi-foci imaging system.

A Hybrid Bionic Image Sensor Achieving FOV Extension and Foveated Imaging

Based on bionic compound eye and human foveated imaging mechanisms, a hybrid bionic image sensor (HBIS) is proposed in this paper to extend the field of view (FOV) with high resolution. First, the hybrid bionic imaging model was developed and the structure parameters of the HBIS were deduced. Second, the properties of the HBIS were simulated, including FOV extension, super-resolution imaging, foveal ratio and so on. Third, a prototype of the HBIS was developed to validate the theory. Imaging experiments were carried out, and the results are in accordance with the simulations, proving the potential of the HBIS for large FOV and high-resolution imaging with low cost.

Low-cost dynamic real-time foveated imager

Foveated imaging systems have the ability to capture local high-resolution or high-magnification images with wide field of view (FOV); thus, they have great potential for applications in the field of monitoring and remote sensing of unmanned aerial vehicles. Hence, foveated optical systems are in strong demand. However, the existing foveated imaging systems either are equipped with expensive modulators or require fixing the local high resolution imaging field, which is not suitable for mass production or object tracking in industrial applications. We propose a low-cost dynamic real-time foveated imaging system for extensive use in the listed applications. Specifically, we place a microlens behind the first intermediary image plane to modulate the local focal length, constructing a local high magnification imaging channel. One two-axis translation stage drives the microlens to scan in the plane perpendicular to the optical axis, resulting in dynamic local high magnifying imaging. Furthermore, the peripheral imaging channel and the foveated imaging channel focus on the same detector, and the post image fusion is unnecessary; the system consists of only a common aspherical lens and thus is very inexpensive. The experimental system has a focal length of 25 mm, a full FOV of 30°, and an entrance pupil diameter of 5 mm, while the local high magnifying imaging channel has a focal length of 35 mm and FOV of 15°. Experiment results show that the low-cost dynamic real-time foveated imaging system performs very well.

Autofocusing imaging system based on laser ranging and a retina-like sample

A novel autofocusing method combining active and passive techniques is proposed to improve autofocusing performance. The proposed method is divided into coarse and fine autofocusing stages. First, a laser ranging finder viewed as the active technique is used to obtain a reasonable initial position of the compensating lens group during the coarse autofocusing stage. Second, a retina-like sampling method is used to decrease the volume of processing data during the fine autofocusing stage. We developed a prototype and performed comparative experiments using different targets based on the proposed method and the traditional passive method (TPM). Under optical power of 9×, the TPM consumed 2.5 s, while the proposed method performed at 1.3 s. The comparative experiments also showed that the proposed method exhibited a better anti-interference property than the traditional method.

Electrically tunable fluidic lens imaging system for laparoscopic fluorescence-guided surgery

The addition of fluorescence guidance in laparoscopic procedures has gained significant interest in recent years, particularly through the use of near infrared (NIR) markers. In this work we present a novel laparoscope camera coupler based on an electrically tunable fluidic lens that permits programmable focus control and has desirable achromatic performance from the visible to the NIR. Its use extends the lower working distance limit and improves detection sensitivity, important for work with molecularly targeted fluorescence markers. We demonstrate its superior optical performance in laparoscopic fluorescence-guided surgery. In vivo results using a tumor specific molecular probe and a nonspecific NIR dye are presented.

Modeling and simulations on retina-like sensors based on curved surface

A space-variant lens array on a curved surface (SVLACS) with a large field of view is proposed to decrease the size and improve the performance of a space-variant lens array on a plane (SVLAP). The whole mathematical models are developed and tested, and comparative simulations between SVLACS and SVLAP are carried out. Under the identical simulated situations, the radius of SVLACS decreases to 58% of SVLAP. Meanwhile, the performance of optical information loss rate is improved from 0.22 to 0.08. The results are beneficial for designing a retina-like image sensor based on SVLACS.