Self-calibration of lensless holographic endoscope using programmable guide stars

Ptycho-endoscopy on a lensless ultrathin fiber bundle tip

Synthetic aperture radar (SAR) utilizes an aircraft-carried antenna to emit electromagnetic pulses and detect the returning echoes. As the aircraft travels across a designated area, it synthesizes a large virtual aperture to improve image resolution. Inspired by SAR, we introduce synthetic aperture ptycho-endoscopy (SAPE) for micro-endoscopic imaging beyond the diffraction limit. SAPE operates by hand-holding a lensless fiber bundle tip to record coherent diffraction patterns from specimens. The fiber cores at the distal tip modulate the diffracted wavefield within a confined area, emulating the role of the 'airborne antenna' in SAR. The handheld operation introduces positional shifts to the tip, analogous to the aircraft's movement. These shifts facilitate the acquisition of a ptychogram and synthesize a large virtual aperture extending beyond the bundle's physical limit. We mitigate the influences of hand motion and fiber bending through a low-rank spatiotemporal decomposition of the bundle's modulation profile. Our tests demonstrate the ability to resolve a 548-nm linewidth on a resolution target. The achieved space-bandwidth product is ~1.1 million effective pixels, representing a 36-fold increase compared to that of the original fiber bundle. Furthermore, SAPE's refocusing capability enables imaging over an extended depth of field exceeding 2 cm. The aperture synthesizing process in SAPE surpasses the diffraction limit set by the probe's maximum collection angle, opening new opportunities for both fiber-based and distal-chip endoscopy in applications such as medical diagnostics and industrial inspection.

Two-wavelength holographic micro-endoscopy

In this paper, we present a method for micro-endoscopic topography measurement utilizing two-wavelength holography. Initially, we evaluate the inter-core dispersion and cross-talk of two commercially available imaging fiber bundles (CFBs) and introduce the concept of virtual surface roughness as a limiting factor of achievable measurement resolution. Subsequently, we describe a micro-endoscope setup incorporating 3D-printed micro-optics, resulting in a total diameter of less than 450 µm. We evaluate the measurement accuracy using a pyramid-shaped test object and demonstrate that a relative measurement error of 7.5% can be achieved with a simple phase unwrapping approach. Moreover, we demonstrate how leveraging a deep learning approach from existing literature, tailored for heavily noisy phase maps, effectively reduces the relative measurement error. The standard deviation of the measurement error is 4.2 times lower with the deep learning approach.

Calibration-free quantitative phase imaging in multi-core fiber endoscopes using end-to-end deep learning

Quantitative phase imaging (QPI) through multi-core fibers (MCFs) has been an emerging in vivo label-free endoscopic imaging modality with minimal invasiveness. However, the computational demands of conventional iterative phase retrieval algorithms have limited their real-time imaging potential. We demonstrate a learning-based MCF phase imaging method that significantly reduced the phase reconstruction time to 5.5 ms, enabling video-rate imaging at 181 fps. Moreover, we introduce an innovative optical system that automatically generated the first, to the best of our knowledge, open-source dataset tailored for MCF phase imaging, comprising 50,176 paired speckles and phase images. Our trained deep neural network (DNN) demonstrates a robust phase reconstruction performance in experiments with a mean fidelity of up to 99.8%. Such an efficient fiber phase imaging approach can broaden the applications of QPI in hard-to-reach areas.

Learned end-to-end high-resolution lensless fiber imaging towards real-time cancer diagnosis

Recent advances in label-free histology promise a new era for real-time diagnosis in neurosurgery. Deep learning using autofluorescence is promising for tumor classification without histochemical staining process. The high image resolution and minimally invasive diagnostics with negligible tissue damage is of great importance. The state of the art is raster scanning endoscopes, but the distal lens optics limits the size. Lensless fiber bundle endoscopy offers both small diameters of a few 100 microns and the suitability as single-use probes, which is beneficial in sterilization. The problem is the inherent honeycomb artifacts of coherent fiber bundles (CFB). For the first time, we demonstrate an end-to-end lensless fiber imaging with exploiting the near-field. The framework includes resolution enhancement and classification networks that use single-shot CFB images to provide both high-resolution imaging and tumor diagnosis. The well-trained resolution enhancement network not only recovers high-resolution features beyond the physical limitations of CFB, but also helps improving tumor recognition rate. Especially for glioblastoma, the resolution enhancement network helps increasing the classification accuracy from 90.8 to 95.6%. The novel technique enables histological real-time imaging with lensless fiber endoscopy and is promising for a quick and minimally invasive intraoperative treatment and cancer diagnosis in neurosurgery.

Real-time holographic lensless micro-endoscopy through flexible fibers via fiber bundle distal holography

Fiber-based micro-endoscopes are a critically important tool for minimally-invasive deep-tissue imaging. However, current micro-endoscopes cannot perform three-dimensional imaging through dynamically-bent fibers without the use of bulky optical elements such as lenses and scanners at the distal end, increasing the footprint and tissue-damage. Great efforts have been invested in developing approaches that avoid distal bulky optical elements. However, the fundamental barrier of dynamic optical wavefront-distortions in propagation through flexible fibers limits current approaches to nearly-static or non-flexible fibers. Here, we present an approach that allows holographic, bend-insensitive, coherence-gated, micro-endoscopic imaging using commercially available multi-core fibers (MCFs). We achieve this by adding a partially-reflecting mirror to the distal fiber-tip, allowing to perform low-coherence full-field phase-shifting holography. We demonstrate widefield diffraction-limited reflection imaging of amplitude and phase targets through dynamically bent fibers at video-rate. Our approach holds potential for label-free investigations of dynamic samples.

Quantitative phase imaging through an ultra-thin lensless fiber endoscope

Quantitative phase imaging (QPI) is a label-free technique providing both morphology and quantitative biophysical information in biomedicine. However, applying such a powerful technique to in vivo pathological diagnosis remains challenging. Multi-core fiber bundles (MCFs) enable ultra-thin probes for in vivo imaging, but current MCF imaging techniques are limited to amplitude imaging modalities. We demonstrate a computational lensless microendoscope that uses an ultra-thin bare MCF to perform quantitative phase imaging with microscale lateral resolution and nanoscale axial sensitivity of the optical path length. The incident complex light field at the measurement side is precisely reconstructed from the far-field speckle pattern at the detection side, enabling digital refocusing in a multi-layer sample without any mechanical movement. The accuracy of the quantitative phase reconstruction is validated by imaging the phase target and hydrogel beads through the MCF. With the proposed imaging modality, three-dimensional imaging of human cancer cells is achieved through the ultra-thin fiber endoscope, promising widespread clinical applications.

Holographic optogenetic stimulation with calcium imaging as an all optical tool for cardiac electrophysiology

All optical approaches to control and read out the electrical activity in a cardiac syncytium can improve our understanding of cardiac electrophysiology. Here, we demonstrate optogenetic stimulation of cardiomyocytes with high spatial precision using light foci generated with a ferroelectric spatial light modulator. Computer generated holograms binarized by bidirectional error diffusion create multiple foci with more even intensity distribution compared with thresholding approach. We evoke the electrical activity of cardiac HL1 cells expressing the channelrhodopsin-2 variant, ChR2(H134R) using single and multiple light foci and at the same time visualize the action potential using a calcium sensitive indicator called Cal-630. We show that localized regions in the cardiac monolayer can be stimulated enabling us to initiate signal propagation from a precise location. Furthermore, we demonstrate that probing the cardiac cells with multiple light foci enhances the excitability of the cardiac network. This approach opens new applications in manipulating and visualizing the electrical activity in a cardiac syncytium.

Real-time complex light field generation through a multi-core fiber with deep learning

The generation of tailored complex light fields with multi-core fiber (MCF) lensless microendoscopes is widely used in biomedicine. However, the computer-generated holograms (CGHs) used for such applications are typically generated by iterative algorithms, which demand high computation effort, limiting advanced applications like fiber-optic cell manipulation. The random and discrete distribution of the fiber cores in an MCF induces strong spatial aliasing to the CGHs, hence, an approach that can rapidly generate tailored CGHs for MCFs is highly demanded. We demonstrate a novel deep neural network-CoreNet, providing accurate tailored CGHs generation for MCFs at a near video rate. The CoreNet is trained by unsupervised learning and speeds up the computation time by two magnitudes with high fidelity light field generation compared to the previously reported CGH algorithms for MCFs. Real-time generated tailored CGHs are on-the-fly loaded to the phase-only spatial light modulator (SLM) for near video-rate complex light fields generation through the MCF microendoscope. This paves the avenue for real-time cell rotation and several further applications that require real-time high-fidelity light delivery in biomedicine.

Benchmarking analysis of computer generated holograms for complex wavefront shaping using pixelated phase modulators

Wavefront shaping with spatial light modulators (SLMs) enables aberration correction, especially for light control through complex media, like biological tissues and multimode fibres. High-fidelity light field shaping is associated with the calculation of computer generated holograms (CGHs), of which there are a variety of algorithms. The achievable performance of CGH algorithms depends on various parameters. In this paper, four different algorithms for CGHs are presented and compared for complex light field generation. Two iterative, double constraint Gerchberg-Saxton and direct search, and the two analytical, superpixel and phase encoding, algorithms are investigated. For each algorithm, a parameter study is performed varying the modulator's pixel number and phase resolution. The analysis refers to mode field generation in multimode fibre endoscopes and communication. This enables generality by generating specific mode combinations according to certain spatial frequency power spectra. Thus, the algorithms are compared varying spatial frequencies applied to different implementation scenarios. Our results demonstrate that the choice of algorithms has a significant impact on the achievable performance. This comprehensive study provides the required guide for CGH algorithm selection, improving holographic systems towards multimode fibre endoscopy and communications.

Memory effect assisted imaging through multimode optical fibres

When light propagates through opaque material, the spatial information it holds becomes scrambled, but not necessarily lost. Two classes of techniques have emerged to recover this information: methods relying on optical memory effects, and transmission matrix (TM) approaches. Here we develop a general framework describing the nature of memory effects in structures of arbitrary geometry. We show how this framework, when combined with wavefront shaping driven by feedback from a guide-star, enables estimation of the TM of any such system. This highlights that guide-star assisted imaging is possible regardless of the type of memory effect a scatterer exhibits. We apply this concept to multimode fibres (MMFs) and identify a 'quasi-radial' memory effect. This allows the TM of an MMF to be approximated from only one end - an important step for micro-endoscopy. Our work broadens the applications of memory effects to a range of novel imaging and optical communication scenarios.

Distortion correction for particle image velocimetry using multiple-input deep convolutional neural network and Hartmann-Shack sensing

Aberrations degrade the accuracy of quantitative, imaging-based measurements, like particle image velocimetry (PIV). Adaptive optical elements can in principle correct the wavefront distortions, but are limited by their technical specifications. Here we propose an actuator-free correction based on a multiple-input deep convolutional neural network which uses an additional input from a wavefront sensor to correct time-varying distortions. It is applied for imaging flow velocimetry to conduct measurements through a fluctuating air-water phase boundary. Dataset for neural network is generated by an experimental setup with a deformable mirror. Correction performance of trained model is estimated in terms of image quality, which is improved significantly, and flow measurement results, where the errors induced by the distortion from fluctuating phase boundary can be corrected by 82 %. The technique has the potential to replace classical closed-loop adaptive optical systems where the performance of the actuators is not sufficient.

Rapid computational cell-rotation around arbitrary axes in 3D with multi-core fiber

Optical trapping is a vital tool in biology, allowing precise optical manipulation of nanoparticles, micro-robots, and cells. Due to the low risk of photodamage and high trap stiffness, fiber-based dual-beam traps are widely used for optical manipulation of large cells. Besides trapping, advanced applications like 3D refractive index tomography need a rotation of cells, which requires precise control of the forces, for example, the acting-point of the forces and the intensities in the region of interest (ROI). A precise rotation of large cells in 3D about arbitrary axes has not been reported yet in dual-beam traps. We introduce a novel dual-beam optical trap in which a multi-core fiber (MCF) is transformed to a phased array, using wavefront shaping and computationally programmable light. The light-field distribution in the trapping region is holographically controlled within 0.1 s, which determines the orientation and the rotation axis of the cell with small retardation. We demonstrate real-time controlled rotation of HL60 cells about all 3D axes with a very high degree of freedom by holographic controlled light through an MCF with a resolution close to the diffraction limit. For the first time, the orientation of the cell can be precisely controlled about all 3D axes in a dual-beam trap. MCFs provide much higher flexibility beyond the bulky optics, enabling lab-on-a-chip applications and can be easily integrated for applications like contactless cell surgery, refractive index tomography, cell-elasticity measurement, which require precise 3D manipulation of cells.

Complex Wavefront Shaping through a Multi-Core Fiber

Wavefront shaping through a multi-core fiber (MCF) is turning into an attractive method for endoscopic imaging and optical cell-manipulation on a chip. However, the discrete distribution and the low number of cores induce pixelated phase modulation, becoming an obstacle for delivering complex light field distributions through MCFs. We demonstrate a novel phase retrieval algorithm named Core–Gerchberg–Saxton (Core-GS) employing the captured core distribution map to retrieve tailored modulation hologram for the targeted intensity distribution at the distal far-field. Complex light fields are reconstructed through MCFs with high fidelity up to 96.2%. Closed-loop control with experimental feedback denotes the capability of the Core-GS algorithm for precise intensity manipulation of the reconstructed light field. Core-GS provides a robust way for wavefront shaping through MCFs; it facilitates the MCF becoming a vital waveguide in endoscopic and lab-on-a-chip applications.

In situ measurement of the isoplanatic patch for imaging through intact bone

Wavefront-shaping (WS) enables imaging through scattering tissues like bone, which is important for neuroscience and bone-regeneration research. WS corrects for the optical aberrations at a given depth and field-of-view (FOV) within the sample; the extent of the validity of which is limited to a region known as the isoplanatic patch (IP). Knowing this parameter helps to estimate the number of corrections needed for WS imaging over a given FOV. In this paper, we first present direct transmissive measurement of murine skull IP using digital optical phase conjugation based focusing. Second, we extend our previously reported phase accumulation ray tracing (PART) method to provide in-situ in-silico estimation of IP, called correlative PART (cPART). Our results show an IP range of 1 to 3 μm for mice within an age range of 8 to 14 days old and 1.00 ± 0.25 μm in a 12-week old adult skull. Consistency between the two measurement approaches indicates that cPART can be used to approximate the IP before a WS experiment, which can be used to calculate the number of corrections required within a given field of view.

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications.

Using fiber-bending-generated speckles for improved working distance and background rejection in lensless micro-endoscopy

Lensless flexible fiber-bundle-based endoscopes allow imaging at depths beyond the reach of conventional microscopes with a minimal footprint. These multicore fibers provide a simple solution for wide-field fluorescent imaging when the target is adjacent to the fiber facet. However, they suffer from a very limited working distance and out-of-focus background. Here, we carefully study the dynamic speckle illumination patterns generated by bending a commercial fiber bundle and show that they can be exploited to allow extended working distance and background rejection, using a super-resolution fluctuations imaging analysis of multiple frames, without the addition of any optical elements.

Video-rate lensless endoscope with self-calibration using wavefront shaping

Lensless fiber endoscopes are of great importance for keyhole imaging. Coherent fiber bundles (CFB) can be used in endoscopes as remote phased arrays to capture images. One challenge is to image at high speed while correcting aberrations induced by the CFB. We propose the combination of digital optical phase conjugation, using a spatial light modulator, with fast scanning, for which a 2D galvo scanner and an adaptive lens are employed. We achieve the transmission of laser and image scanning through the CFB. Video-rate imaging at 20 Hz in 2D with subcellular resolution is demonstrated in 3D with 1 Hz. The sub-millimeter-diameter scanning endoscope has a great potential in biomedicine, for manipulation, e.g., in optogenetics, as well as in imaging.

Deep Learning for Computational Mode Decomposition in Optical Fibers

Multimode fibers are regarded as the key technology for the steady increase in data rates in optical communication. However, light propagation in multimode fibers is complex and can lead to distortions in the transmission of information. Therefore, strategies to control the propagation of light should be developed. These strategies include the measurement of the amplitude and phase of the light field after propagation through the fiber. This is usually done with holographic approaches. In this paper, we discuss the use of a deep neural network to determine the amplitude and phase information from simple intensity-only camera images. A new type of training was developed, which is much more robust and precise than conventional training data designs. We show that the performance of the deep neural network is comparable to digital holography, but requires significantly smaller efforts. The fast characterization of multimode fibers is particularly suitable for high-performance applications like cyberphysical systems in the internet of things.

Structured light beams created through a multimode fiber via virtual Fourier filtering based on digital optical phase conjugation

Digital optical phase conjugation (DOPC) is a newly developed technique in wavefront shaping to control light propagation through complex media. Currently, DOPC has been demonstrated for the reconstruction of two- and three-dimensional targets and enabled important applications in many areas. Nevertheless, the reconstruction results are only phase conjugated to the original input targets. Herein, we demonstrate that DOPC could be further developed for creating structured light beams through a multimode fiber (MMF). By applying annular filtering in the virtual Fourier domain of the acquired speckle field, we realize the creation of the quasi-Bessel and donut beams through the MMF. In principle, arbitrary amplitude and/or phase circular symmetry filtering could be performed in the Fourier domain, thus generating the corresponding point spread functions. We expect that the reported technique can be useful for super-resolution endoscopic imaging and optical manipulation through MMFs.

Diffraction-limited axial scanning in thick biological tissue with an aberration-correcting adaptive lens

Diffraction-limited deep focusing into biological tissue is challenging due to aberrations that lead to a broadening of the focal spot. The diffraction limit can be restored by employing aberration correction for example with a deformable mirror. However, this results in a bulky setup due to the required beam folding. We propose a bi-actuator adaptive lens that simultaneously enables axial scanning and the correction of specimen-induced spherical aberrations with a compact setup. Using the bi-actuator lens in a confocal microscope, we show diffraction-limited axial scanning up to 340 μm deep inside a phantom specimen. The application of this technique to in vivo measurements of zebrafish embryos with reporter-gene-driven fluorescence in a thyroid gland reveals substructures of the thyroid follicles, indicating that the bi-actuator adaptive lens is a meaningful supplement to the existing adaptive optics toolset.

Holographic lensless fiber endoscope with needle size using self-calibration

Endoscopes enable optical keyhole access in many applications for instance in biomedicine. In general, coherent fiber bundles (CFB) are used in conjunction with rigid lens systems which determine a fixed image plane. However, the lens system limits the minimum diameter of the endoscope typically to several millimeters. Additionally, only pixelated two-dimensional amplitude patterns can be transferred due to phase scrambling between adjacent cores. These limitations can be overcome by digital optical elements. Thus, in principle thinner, lensless, holographic endoscopes with a three-dimensional adjustable focus for imaging and illumination can be realized. So far, several techniques based on single mode CFB and multi mode fibers (MMF) have been presented. However, these techniques require access to both sides of the fiber, in order to calibrate the bending and temperature sensitive phase distortion, which is not possible in a real application. We present the feasibility of an in-situ calibration and compensation of a CFB with single sided access. A lensless endoscope with a diameter of only 500 µm, a spatial resolution around 1 µm and video rate capability is realized.

Selective femtosecond laser ablation via two-photon fluorescence imaging through a multimode fiber

We demonstrate the ability of a multimode fiber probe to provide two-photon fluorescence (TPF) imaging feedback that guides the femtosecond laser ablation (FLA) in biological samples for highly selective modifications. We implement the system through the propagation of high power femtosecond pulses through a graded-index (GRIN) multimode fiber and we investigate the limitations posed by the high laser peak intensities required for laser ablation. We demonstrate that the GRIN fiber probe can deliver laser intensities up to 1.5x10¹³ W/cm², sufficient for the ablation of a wide range of materials, including biological samples. Wavefront shaping through an ultrathin probe of around 400 μm in diameter is used for diffraction limited focusing and digital scanning of the focus spot. Selective FLA of cochlear hair cells is performed based on the TPF images obtained through the same multimode fiber probe.

Transmission Matrix Measurement of Multimode Optical Fibers by Mode-Selective Excitation Using One Spatial Light Modulator

Multimode fibers (MMF) are promising candidates to increase the data rate while reducing the space required for optical fiber networks. However, their use is hampered by mode mixing and other effects, leading to speckled output patterns. This can be overcome by measuring the transmission matrix (TM) of a multimode fiber. In this contribution, a mode-selective excitation of complex amplitudes is performed with only one phase-only spatial light modulator. The light field propagating through the fiber is measured holographically and is analyzed by a rapid decomposition method. This technique requires a small amount of measurements N, which corresponds to the degree of freedom of the fiber. The TM determines the amplitude and phase relationships of the modes, which allows us to understand the mode scrambling processes in the MMF and can be used for mode division multiplexing.

Light scattering control in transmission and reflection with neural networks

Scattering often limits the controlled delivery of light in applications such as biomedical imaging, optogenetics, optical trapping, and fiber-optic communication or imaging. Such scattering can be controlled by appropriately shaping the light wavefront entering the material. Here, we develop a machine-learning approach for light control. Using pairs of binary intensity patterns and intensity measurements we train neural networks (NNs) to provide the wavefront corrections necessary to shape the beam after the scatterer. Additionally, we demonstrate that NNs can be used to find a functional relationship between transmitted and reflected speckle patterns. Establishing the validity of this relationship, we focus and scan in transmission through opaque media using reflected light. Our approach shows the versatility of NNs for light shaping, for efficiently and flexibly correcting for scattering, and in particular the feasibility of transmission control based on reflected light.