Real-time holographic lensless micro-endoscopy through flexible fibers via fiber bundle distal holography

Resolution-enhanced multi-core fiber imaging learned on a digital twin for cancer diagnosis

Significance

Deep learning enables label-free all-optical biopsies and automated tissue classification. Endoscopic systems provide intraoperative diagnostics to deep tissue and speed up treatment without harmful tissue removal. However, conventional multi-core fiber (MCF) endoscopes suffer from low resolution and artifacts, which hinder tumor diagnostics.

Aim

We introduce a method to enable unpixelated, high-resolution tumor imaging through a given MCF with a diameter of around 0.65 mm and arbitrary core arrangement and inhomogeneous transmissivity.

Approach

Image reconstruction is based on deep learning and the digital twin concept of the single-reference-based simulation with inhomogeneous optical properties of MCF and transfer learning on a small experimental dataset of biological tissue. The reference provided physical information about the MCF during the training processes.

Results

For the simulated data, hallucination caused by the MCF inhomogeneity was eliminated, and the averaged peak signal-to-noise ratio and structural similarity were increased from 11.2 dB and 0.20 to 23.4 dB and 0.74, respectively. By transfer learning, the metrics of independent test images experimentally acquired on glioblastoma tissue ex vivo can reach up to 31.6 dB and 0.97 with 14 fps computing speed.

Conclusions

With the proposed approach, a single reference image was required in the pre-training stage and laborious acquisition of training data was bypassed. Validation on glioblastoma cryosections with transfer learning on only 50 image pairs showed the capability for high-resolution deep tissue retrieval and high clinical feasibility.

Advancing the path to in-vivo imaging in freely moving mice via multimode-multicore fiber based holographic endoscopy

Significance

Hair-thin multimode optical fiber-based holographic endoscopes have gained considerable interest in modern neuroscience for their ability to achieve cellular and even subcellular resolution during in-vivo deep brain imaging. However, the application of multimode fibers in freely moving animals presents a persistent challenge as it is difficult to maintain optimal imaging performance while the fiber undergoes deformations.

Aim

We propose a fiber solution for challenging in-vivo applications with the capability of deep brain high spatial resolution imaging and neuronal activity monitoring in anesthetized as well as awake behaving mice.

Approach

We used our previously developed M 3 CF multimode-multicore fiber to record fluorescently labeled neurons in anesthetized mice. Our M 3 CF exhibits a cascaded refractive index structure, enabling two distinct regimes of light transport that imitate either a multimode or a multicore fiber. The M 3 CF has been specifically designed for use in the initial phase of an in-vivo experiment, allowing for the navigation of the endoscope's distal end toward the targeted brain structure. The multicore regime enables the transfer of light to and from each individual neuron within the field of view. For chronic experiments in awake behaving mice, it is crucial to allow for disconnecting the fiber and the animal between experiments. Therefore, we provide here an effective solution and establish a protocol for reconnection of two segments of M 3 CF with hexagonally arranged corelets.

Results

We successfully utilized the M 3 CF to image neurons in anaesthetized transgenic mice expressing enhanced green fluorescent protein. Additionally, we compared imaging results obtained with the M 3 CF with larger numerical aperture (NA) fibers in fixed whole-brain tissue.

Conclusions

This study focuses on addressing challenges and providing insights into the use of multimode-multicore fibers as imaging solutions for in-vivo applications. We suggest that the upcoming version of the M 3 CF increases the overall NA between the two cladding layers to allow for access to high resolution spatial imaging. As the NA increases in the multimode regime, the fiber diameter and ring structure must be reduced to minimize the computational burden and invasiveness.

Honeycomb effect elimination in differential phase fiber-bundle-based endoscopy

Fiber-bundle-based endoscopy, with its ultrathin probe and micrometer-level resolution, has become a widely adopted imaging modality for in vivo imaging. However, the fiber bundles introduce a significant honeycomb effect, primarily due to the multi-core structure and crosstalk of adjacent fiber cores, which superposes the honeycomb pattern image on the original image. To tackle this issue, we propose an iterative-free spatial pixel shifting (SPS) algorithm, designed to suppress the honeycomb effect and enhance real-time imaging performance. The process involves the creation of three additional sub-images by shifting the original image by one pixel at 0, 45, and 90 degree angles. These four sub-images are then used to compute differential maps in the x and y directions. By performing spiral integration on these differential maps, we reconstruct a honeycomb-free image with improved details. Our simulations and experimental results, conducted on a self-built fiber bundle-based endoscopy system, demonstrate the effectiveness of the SPS algorithm. SPS significantly improves the image quality of reflective objects and unlabeled transparent scattered objects, laying a solid foundation for biomedical endoscopic applications.

Ptychographic lensless coherent endomicroscopy through a flexible fiber bundle

Conventional fiber-bundle-based endoscopes allow minimally invasive imaging through flexible multi-core fiber (MCF) bundles by placing a miniature lens at the distal tip and using each core as an imaging pixel. In recent years, lensless imaging through MCFs was made possible by correcting the core-to-core phase distortions pre-measured in a calibration procedure. However, temporally varying wavefront distortions, for instance, due to dynamic fiber bending, pose a challenge for such approaches. Here, we demonstrate a coherent lensless imaging technique based on intensity-only measurements insensitive to core-to-core phase distortions. We leverage a ptychographic reconstruction algorithm to retrieve the phase and amplitude profiles of reflective objects placed at a distance from the fiber tip, using as input a set of diffracted intensity patterns reflected from the object when the illumination is scanned over the MCF cores. Our approach thus utilizes an acquisition process equivalent to confocal microendoscopy, only replacing the single detector with a camera.

Improved resolution in fiber bundle inline holographic microscopy using multiple illumination sources

Recent work has shown that high-quality inline holographic microscopy images can be captured through fiber imaging bundles. Speckle patterns arising from modal interference within the bundle cores can be minimized by use of a partially-coherent optical source such as an LED delivered via a multimode fiber. This allows numerical refocusing of holograms from samples at working distances of up to approximately 1 mm from the fiber bundle before the finite coherence begins to degrade the lateral resolution. However, at short working distances the lateral resolution is limited not by coherence, but by sampling effects due to core-to-core spacing in the bundle. In this article we demonstrate that multiple shifted holograms can be combined to improve the resolution by a factor of two. The shifted holograms can be rapidly acquired by sequentially firing LEDs, which are each coupled to their own, mutually offset, illumination fiber. Following a one-time calibration, resolution-enhanced images are created in real-time at an equivalent net frame rate of up to 7.5 Hz. The resolution improvement is demonstrated quantitatively using a resolution target and qualitatively using mounted biological slides. At longer working distances, beyond 0.6 mm, the improvement is reduced as resolution becomes limited by the source spatial and temporal coherence.

Calibration-free quantitative phase imaging in multi-core fiber endoscopes using end-to-end deep learning

Quantitative phase imaging (QPI) through multi-core fibers (MCFs) has been an emerging in vivo label-free endoscopic imaging modality with minimal invasiveness. However, the computational demands of conventional iterative phase retrieval algorithms have limited their real-time imaging potential. We demonstrate a learning-based MCF phase imaging method that significantly reduced the phase reconstruction time to 5.5 ms, enabling video-rate imaging at 181 fps. Moreover, we introduce an innovative optical system that automatically generated the first, to the best of our knowledge, open-source dataset tailored for MCF phase imaging, comprising 50,176 paired speckles and phase images. Our trained deep neural network (DNN) demonstrates a robust phase reconstruction performance in experiments with a mean fidelity of up to 99.8%. Such an efficient fiber phase imaging approach can broaden the applications of QPI in hard-to-reach areas.

Lensless single-fiber ghost imaging

We demonstrate lensless single-fiber ghost imaging, which allows illumination and collection using a single optical fiber without a transmission-type system. Speckle patterns with relative coincidence degrees of 0.14 were formed by image reconstruction using improved differential ghost imaging. Employing fiber with a diameter of 105 µm, we achieved a spatial resolution of 0.05 mm in an observing area of 9m m 2, at a working distance of 10 mm. Compared to a conventional neuroendoscope at a power density of 94m W/c m 2, our imaging could be realized by extremely weak illumination at a laser power density of 0.10m W/c m 2. Using our lensless single-fiber ghost imaging, with 30,000 speckle patterns and implementing a diffuser, we attained an average coincidence degree of 0.45.

Optimizing the internal phase reference to shape the output of a multimode optical fiber

Pre-shaping light to achieve desired amplitude distributions at the tip of a multimode fiber (MMF) has emerged as a powerful method allowing a wide range of imaging techniques to be implemented at the distal facet. Such techniques rely on measuring the transmission matrix of the optically turbid waveguide which scrambles the coherent input light into an effectively random speckle pattern. Typically, this is done by measuring the interferogram between the output speckle and a reference beam. In recent years, an optical setup where the reference beam passes through the MMF has become an attractive configuration because of the high interferometric stability of the common optical path. However, the merits and drawbacks of an internal reference beam remain controversial. The measurement of the transmission matrix is known to depend on the choice of internal reference and has been reported to result in "blind spots" due to phase singularities of the reference beam. Here, we describe how the focussing efficiency of the calibration can be increased by several percent by optimising the choice of internal reference beam.

Fourier holographic endoscopy for imaging continuously moving objects

Coherent fiber bundles are widely used for endoscopy, but conventional approaches require distal optics to form an object image and acquire pixelated information owing to the geometry of the fiber cores. Recently, holographic recording of a reflection matrix enables a bare fiber bundle to perform pixelation-free microscopic imaging as well as allows a flexible mode operation, because the random core-to-core phase retardations due to any fiber bending and twisting could be removed in situ from the recorded matrix. Despite its flexibility, the method is not suitable for a moving object because the fiber probe should remain stationary during the matrix recording to avoid the alteration of the phase retardations. Here, we acquire a reflection matrix of a Fourier holographic endoscope equipped with a fiber bundle and explore the effect of fiber bending on the recorded matrix. By removing the motion effect, we develop a method that can resolve the perturbation of the reflection matrix caused by a continuously moving fiber bundle. Thus, we demonstrate high-resolution endoscopic imaging through a fiber bundle, even when the fiber probe changes its shape along with the moving objects. The proposed method can be used for minimally invasive monitoring of behaving animals.