1
|
Wang J, Chen C, You W, Jiao Y, Liu X, Jiang X, Lu W. Honeycomb effect elimination in differential phase fiber-bundle-based endoscopy. OPTICS EXPRESS 2024; 32:20682-20694. [PMID: 38859444 DOI: 10.1364/oe.526033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2024] [Accepted: 05/10/2024] [Indexed: 06/12/2024]
Abstract
Fiber-bundle-based endoscopy, with its ultrathin probe and micrometer-level resolution, has become a widely adopted imaging modality for in vivo imaging. However, the fiber bundles introduce a significant honeycomb effect, primarily due to the multi-core structure and crosstalk of adjacent fiber cores, which superposes the honeycomb pattern image on the original image. To tackle this issue, we propose an iterative-free spatial pixel shifting (SPS) algorithm, designed to suppress the honeycomb effect and enhance real-time imaging performance. The process involves the creation of three additional sub-images by shifting the original image by one pixel at 0, 45, and 90 degree angles. These four sub-images are then used to compute differential maps in the x and y directions. By performing spiral integration on these differential maps, we reconstruct a honeycomb-free image with improved details. Our simulations and experimental results, conducted on a self-built fiber bundle-based endoscopy system, demonstrate the effectiveness of the SPS algorithm. SPS significantly improves the image quality of reflective objects and unlabeled transparent scattered objects, laying a solid foundation for biomedical endoscopic applications.
Collapse
|
2
|
Jiang Z, Wen Y, Song L, Li D, Zhao X. Optical fiber bundle differential compressive imaging. OPTICS LETTERS 2024; 49:2297-2300. [PMID: 38691703 DOI: 10.1364/ol.519161] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Accepted: 03/31/2024] [Indexed: 05/03/2024]
Abstract
We present a differential compressive imaging method for an optical fiber bundle (OFB), which provides a solution for an ultrathin bend-resistant endoscope with high resolution. This method uses an OFB and a diffuser to generate speckle illumination patterns. Differential operation is additionally applied to the speckle patterns to produce sensing matrices, by which the correlation between the matrices is greatly reduced from 0.875 to 0.0275, which ensures the high quality of image reconstruction. Pixilation artifacts from the fiber core arrangement are also effectively eliminated with this configuration. We demonstrate high-resolution reconstruction of images of 132 × 132 pixels with a compression rate of 12% using 77 fiber cores, the total diameter of which is only about 91 µm. An experimental verification proves that this method is tolerant to a limited degree of fiber bending, which provides a potential approach for robust high-resolution fiber endoscopy.
Collapse
|
3
|
Saiham D, Zhu Z, Klein AB, Pang SS. Accelerated fixed-point iterative reconstruction for fiber borescope imaging. OPTICS EXPRESS 2023; 31:38355-38364. [PMID: 38017943 DOI: 10.1364/oe.495252] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Accepted: 10/11/2023] [Indexed: 11/30/2023]
Abstract
Computational imaging systems with embedded processing have potential advantages in power consumption, computing speed, and cost. However, common processors in embedded vision systems have limited computing capacity and low level of parallelism. The widely used iterative algorithms for image reconstruction rely on floating-point processors to ensure calculation precision, which require more computing resources than fixed-point processors. Here we present a regularized Landweber fixed-point iterative solver for image reconstruction, implemented on a field programmable gated array (FPGA). Compared with floating-point embedded uniprocessors, iterative solvers implemented on the fixed-point FPGA gain 1 to 2 orders of magnitude acceleration, while achieving the same reconstruction accuracy in comparable number of effective iterations. Specifically, we have demonstrated the proposed fixed-point iterative solver in fiber borescope image reconstruction, successfully correcting the artifacts introduced by the lenses and fiber bundle.
Collapse
|
4
|
Jiang Z, Zhao X, Wen Y, Peng Q, Li D, Song L. Block-based compressed sensing for fast optic fiber bundle imaging with high spatial resolution. OPTICS EXPRESS 2023; 31:17235-17249. [PMID: 37381463 DOI: 10.1364/oe.488171] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 04/23/2023] [Indexed: 06/30/2023]
Abstract
The resolution of traditional fiber bundle imaging is usually limited by the density and the diameter of the fiber cores. To improve the resolution, compression sensing was introduced to resolve multiple pixels from a single fiber core, but current methods have the drawbacks of excessive sampling and long reconstruction time. In this paper, we present, what we believe to be, a novel block-based compressed sensing scheme for fast realization of high-resolution optic fiber bundle imaging. In this method, the target image is segmented into multiple small blocks, each of which covers the projection area of one fiber core. All block images are independently and simultaneously sampled and the intensities are recorded by a two-dimensional detector after they are collected and transmitted through corresponding fiber cores. Because the size of sampling patterns and the sampling numbers are greatly reduced, the reconstruction complexity and reconstruction time are also decreased. According to the simulation analysis, our method is 23 times faster than the current compressed sensing optical fiber imaging for reconstructing a fiber image of 128 × 128 pixels, while the sampling number is only 0.39%. Experiment results demonstrate that the method is also effective for reconstructing large target images and the number of sampling does not increase with the size of the image. Our finding may provide a new idea for high-resolution real-time imaging of fiber bundle endoscope.
Collapse
|
5
|
Dumas JP, Lodhi MA, Bajwa WU, Pierce MC. Computational imaging with spectral coding increases the spatial resolution of fiber optic bundles. OPTICS LETTERS 2023; 48:1088-1091. [PMID: 36857220 DOI: 10.1364/ol.477579] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Accepted: 01/22/2023] [Indexed: 06/18/2023]
Abstract
Fiber optic bundles are used in narrow-diameter medical and industrial instruments for acquiring images from confined locations. Images transmitted through these bundles contain only one pixel of information per fiber core and fail to capture information from the cladding region between cores. Both factors limit the spatial resolution attainable with fiber bundles. We show here that computational imaging (CI) can be combined with spectral coding to overcome these two fundamental limitations and improve spatial resolution in fiber bundle imaging. By acquiring multiple images of a scene with a high-resolution mask pattern imposed, up to 17 pixels of information can be recovered from each fiber core. A dispersive element at the distal end of the bundle imparts a wavelength-dependent lateral shift on light from the object. This enables light that would otherwise be lost at the inter-fiber cladding to be transmitted through adjacent fiber cores. We experimentally demonstrate this approach using synthetic and real objects. Using CI with spectral coding, object features 5× smaller than individual fiber cores were resolved, whereas conventional imaging could only resolve features at least 1.5× larger than each core. In summary, CI combined with spectral coding provides an approach for overcoming the two fundamental limitations of fiber optic bundle imaging.
Collapse
|
6
|
Kim E, Kim S, Choi M, Seo T, Yang S. Honeycomb Artifact Removal Using Convolutional Neural Network for Fiber Bundle Imaging. SENSORS (BASEL, SWITZERLAND) 2022; 23:333. [PMID: 36616931 PMCID: PMC9824069 DOI: 10.3390/s23010333] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Revised: 12/14/2022] [Accepted: 12/24/2022] [Indexed: 06/17/2023]
Abstract
We present a new deep learning framework for removing honeycomb artifacts yielded by optical path blocking of cladding layers in fiber bundle imaging. The proposed framework, HAR-CNN, provides an end-to-end mapping from a raw fiber bundle image to an artifact-free image via a convolution neural network (CNN). The synthesis of honeycomb patterns on ordinary images allows conveniently learning and validating the network without the enormous ground truth collection by extra hardware setups. As a result, HAR-CNN shows significant performance improvement in honeycomb pattern removal and also detailed preservation for the 1961 USAF chart sample, compared with other conventional methods. Finally, HAR-CNN is GPU-accelerated for real-time processing and enhanced image mosaicking performance.
Collapse
Affiliation(s)
- Eunchan Kim
- Center for Intelligent and Interactive Robotics, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea
- Department of Mechanical Convergence Engineering, Hanyang University, Seoul 04763, Republic of Korea
| | - Seonghoon Kim
- Department of Biological Sciences, Seoul National University, Seoul 03080, Republic of Korea
| | - Myunghwan Choi
- Department of Biological Sciences, Seoul National University, Seoul 03080, Republic of Korea
| | - Taewon Seo
- Department of Mechanical Convergence Engineering, Hanyang University, Seoul 04763, Republic of Korea
| | - Sungwook Yang
- Center for Intelligent and Interactive Robotics, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea
| |
Collapse
|
7
|
Wisotzky EL, Kossack B, Uecker FC, Arens P, Hilsmann A, Eisert P. Validation of two techniques for intraoperative hyperspectral human tissue determination. J Med Imaging (Bellingham) 2020; 7:065001. [PMID: 33241074 PMCID: PMC7675006 DOI: 10.1117/1.jmi.7.6.065001] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2020] [Accepted: 10/26/2020] [Indexed: 12/23/2022] Open
Abstract
Purpose: Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. Our work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Approach: Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. We introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filter-wheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 up to 975 nm. They are calibrated and validated using the same database and calibration set. Results: For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus the results acquired with both settings become transferable and comparable to each other as well as between different interventions. On patient data of two different otorhinolaryngology procedures, we analyze the optical behaviors of different soft tissues and show a visualization of such different spectral information. Conclusion: The introduced calibration pipeline for different HSI setups allows comparison between all acquired spectral information. Clinical in vivo data underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both introduced setups. Thereby, we demonstrate their feasibility for the in vivo analysis and categorization of different human soft tissues.
Collapse
Affiliation(s)
- Eric L. Wisotzky
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
- Humboldt-Universität zu Berlin, Visual Computing Group, Berlin, Germany
| | - Benjamin Kossack
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
| | - Florian C. Uecker
- Charité—Universitätsmedizin Berlin, Department of Otorhinolaryngology, Berlin, Germany
| | - Philipp Arens
- Charité—Universitätsmedizin Berlin, Department of Otorhinolaryngology, Berlin, Germany
| | - Anna Hilsmann
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
| | - Peter Eisert
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
- Humboldt-Universität zu Berlin, Visual Computing Group, Berlin, Germany
| |
Collapse
|
8
|
Wisotzky EL, Kossack B, Uecker FC, Arens P, Hilsmann A, Eisert P. Validation of two techniques for intraoperative hyperspectral human tissue determination. J Med Imaging (Bellingham) 2020; 7:065001. [PMID: 33241074 DOI: 10.1117/12.251281] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2020] [Accepted: 10/26/2020] [Indexed: 05/22/2023] Open
Abstract
Purpose: Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. Our work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Approach: Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. We introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filter-wheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 up to 975 nm. They are calibrated and validated using the same database and calibration set. Results: For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus the results acquired with both settings become transferable and comparable to each other as well as between different interventions. On patient data of two different otorhinolaryngology procedures, we analyze the optical behaviors of different soft tissues and show a visualization of such different spectral information. Conclusion: The introduced calibration pipeline for different HSI setups allows comparison between all acquired spectral information. Clinical in vivo data underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both introduced setups. Thereby, we demonstrate their feasibility for the in vivo analysis and categorization of different human soft tissues.
Collapse
Affiliation(s)
- Eric L Wisotzky
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
- Humboldt-Universität zu Berlin, Visual Computing Group, Berlin, Germany
| | - Benjamin Kossack
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
| | - Florian C Uecker
- Charité-Universitätsmedizin Berlin, Department of Otorhinolaryngology, Berlin, Germany
| | - Philipp Arens
- Charité-Universitätsmedizin Berlin, Department of Otorhinolaryngology, Berlin, Germany
| | - Anna Hilsmann
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
| | - Peter Eisert
- Fraunhofer Heinrich Hertz Institute, Computer Vision and Graphics Group, Berlin, Germany
- Humboldt-Universität zu Berlin, Visual Computing Group, Berlin, Germany
| |
Collapse
|
9
|
Clancy NT, Jones G, Maier-Hein L, Elson DS, Stoyanov D. Surgical spectral imaging. Med Image Anal 2020; 63:101699. [PMID: 32375102 PMCID: PMC7903143 DOI: 10.1016/j.media.2020.101699] [Citation(s) in RCA: 62] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2019] [Revised: 03/30/2020] [Accepted: 04/06/2020] [Indexed: 12/24/2022]
Abstract
Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.
Collapse
Affiliation(s)
- Neil T Clancy
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, United Kingdom; Centre for Medical Image Computing (CMIC), Department of Medical Physics and Biomedical Engineering, University College London, United Kingdom.
| | - Geoffrey Jones
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, United Kingdom; Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, United Kingdom
| | | | - Daniel S Elson
- Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London, United Kingdom; Department of Surgery and Cancer, Imperial College London, United Kingdom
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, United Kingdom; Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, United Kingdom
| |
Collapse
|
10
|
Waterhouse DJ, Luthman AS, Yoon J, Gordon GSD, Bohndiek SE. Quantitative evaluation of comb-structure correction methods for multispectral fibrescopic imaging. Sci Rep 2018; 8:17801. [PMID: 30542081 PMCID: PMC6290790 DOI: 10.1038/s41598-018-36088-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2018] [Accepted: 11/13/2018] [Indexed: 02/07/2023] Open
Abstract
Removing the comb artifact introduced by imaging fibre bundles, or 'fibrescopes', for example in medical endoscopy, is essential to provide high quality images to the observer. Multispectral imaging (MSI) is an emerging method that combines morphological (spatial) and chemical (spectral) information in a single data 'cube'. When a fibrescope is coupled to a spectrally resolved detector array (SRDA) to perform MSI, comb removal is complicated by the demosaicking step required to reconstruct the multispectral data cube. To understand the potential for using SRDAs as multispectral imaging sensors in medical endoscopy, we assessed five comb correction methods with respect to five performance metrics relevant to biomedical imaging applications: processing time, resolution, smoothness, signal and the accuracy of spectral reconstruction. By assigning weights to each metric, which are determined by the particular imaging application, our results can be used to select the correction method to achieve best overall performance. In most cases, interpolation gave the best compromise between the different performance metrics when imaging using an SRDA.
Collapse
Affiliation(s)
- Dale J Waterhouse
- Department of Physics, University of Cambridge, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, CB2 0RE, UK
| | - A Siri Luthman
- Department of Physics, University of Cambridge, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, CB2 0RE, UK
| | - Jonghee Yoon
- Department of Physics, University of Cambridge, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, CB2 0RE, UK
| | - George S D Gordon
- Department of Physics, University of Cambridge, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, CB2 0RE, UK
- Department of Engineering, University of Cambridge, Cambridge, CB3 0FA, UK
| | - Sarah E Bohndiek
- Department of Physics, University of Cambridge, Cambridge, CB3 0HE, UK.
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, CB2 0RE, UK.
| |
Collapse
|
11
|
Dvornikov A, Gratton E. Hyperspectral imaging in highly scattering media by the spectral phasor approach using two filters. BIOMEDICAL OPTICS EXPRESS 2018; 9:3503-3511. [PMID: 30338135 PMCID: PMC6191637 DOI: 10.1364/boe.9.003503] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/23/2018] [Revised: 06/14/2018] [Accepted: 06/18/2018] [Indexed: 05/25/2023]
Abstract
Hyperspectral imaging is a common technique in fluorescence microscopy to obtain the emission spectrum at each pixel of an image. However, methods to obtain spectral resolution based on diffraction gratings or integrated prisms work poorly when the sample is strongly scattering. We developed a microscope named the DIVER that collects the fluorescence emission over a very large angle. Since the fluorescence light after passing through the multiple scattering sample is not collimated, the use of grating or prisms strongly limits the amount of light that can be used with available hyperspectral devices. Here we show that 2 filters that accept uncollimated light over a large aperture are sufficient to calculate the spectral phasor rather than displaying the entire spectrum. Using the properties of the spectral phasors, we can resolve spectral components and perform the type of data analyses that are usually performed in hyperspectral image analysis.
Collapse
Affiliation(s)
- Alexander Dvornikov
- Laboratory for Fluorescence Dynamics, Department of Biomedical Engineering, University of California Irvine, Irvine, CA 92697, USA
| | - Enrico Gratton
- Laboratory for Fluorescence Dynamics, Department of Biomedical Engineering, University of California Irvine, Irvine, CA 92697, USA
| |
Collapse
|