1
|
Zhou J, Dong X, Liu Q. Context-aware dynamic filtering network for confocal laser endomicroscopy image denoising. Phys Med Biol 2023; 68:195014. [PMID: 37647912 DOI: 10.1088/1361-6560/acf558] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 08/30/2023] [Indexed: 09/01/2023]
Abstract
Objective.As an emerging diagnosis technology for gastrointestinal diseases, confocal laser endomicroscopy (CLE) is limited by the physical structure of the fiber bundle, leading to the inevitable production of various forms of noise during the imaging process. However, existing denoising methods based on hand-crafted features inefficiently deal with realistic noise in CLE images. To alleviate this challenge, we proposed context-aware kernel estimation and multi-scale dynamic fusion modules to remove realistic noise in CLE images, including multiplicative and additive white noise.Approach.Specifically, a realistic noise statistics model with random noise specific to CLE data is constructed and further used to develop a self-supervised denoised model without the participation of clean images. Secondly, context-aware kernel estimation, which improves the representation of features by similar learnable region weights, addresses the problem of the non-uniform distribution of noises in CLE images and proposes a lightweight denoised model (CLENet). Thirdly, we have developed a multi-scale dynamic fusion module that decouples and recalibrates features, providing a precise and contextually enriched representation of features. Finally, we integrated two developed modules into a U-shaped backbone to build an efficient denoising network named U-CLENet.Main Results.Both proposed methods achieve comparable or better performance with low computational complexity on two gastrointestinal disease CLE image datasets using the same training benchmark.Significance.The proposed approaches improve the visual quality of unclear CLE images for various stages of tumor development, helping to reduce the rate of misdiagnosis in clinical decision-making and achieve computer graphics-assisted diagnosis.
Collapse
Affiliation(s)
- Jingjun Zhou
- School of Biomedical Engineering, Hainan University, 570228 Haikou, People's Republic of China
| | - Xiangjiang Dong
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, 430074 Wuhan, People's Republic of China
| | - Qian Liu
- School of Biomedical Engineering, Hainan University, 570228 Haikou, People's Republic of China
- Key Laboratory of Biomedical Engineering of Hainan Province, School of Biomedical Engineering, Hainan University, 570228 Haikou, People's Republic of China
| |
Collapse
|
2
|
Zhang C, Gu Y, Yang GZ. Contrastive Adversarial Learning for Endomicroscopy Imaging Super-Resolution. IEEE J Biomed Health Inform 2023; 27:3994-4005. [PMID: 37171919 DOI: 10.1109/jbhi.2023.3275563] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
Endomicroscopy is an emerging imaging modality for real-time optical biopsy. One limitation of existing endomicroscopy based on coherent fibre bundles is that the image resolution is intrinsically limited by the number of fibres that can be practically integrated within the small imaging probe. To improve the image resolution, Super-Resolution (SR) techniques combined with image priors can enhance the clinical utility of endomicroscopy whereas existing SR algorithms suffer from the lack of explicit guidance from ground truth high-resolution (HR) images. In this article, we propose an unsupervised SR pipeline to allow stable offline and kernel-generic learning. Our method takes advantage of both internal statistics and external cross-modality priors. To improve the joint learning process, we present a Sharpness-aware Contrastive Generative Adversarial Network (SCGAN) with two dedicated modules, a sharpness-aware generator and a contrastive-learning discriminator. In the generator, an auxiliary task of sharpness discrimination is formulated to facilitate internal learning by considering the rankings of training instances in various sharpness levels. In the discriminator, we design a contrastive-learning module to mitigate the ill-posed nature of SR tasks via constraints from both positive and negative images. Experiments on multiple datasets demonstrate that SCGAN reduces the performance gap between previous unsupervised approaches and the upper bounds defined in supervised settings by more than 50%, delivering a new state-of-the-art performance score for endomicroscopy super-resolution. Further application on a realistic Voronoi-based pCLE downsampling kernel proves that SCGAN attains PSNR of 35.851 dB, improving 5.23 dB compared with the traditional Delaunay interpolation.
Collapse
|
3
|
Eadie M, Liao J, Ageeli W, Nabi G, Krstajić N. Fiber Bundle Image Reconstruction Using Convolutional Neural Networks and Bundle Rotation in Endomicroscopy. SENSORS (BASEL, SWITZERLAND) 2023; 23:2469. [PMID: 36904673 PMCID: PMC10007631 DOI: 10.3390/s23052469] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 02/15/2023] [Accepted: 02/20/2023] [Indexed: 06/18/2023]
Abstract
Fiber-bundle endomicroscopy has several recognized drawbacks, the most prominent being the honeycomb effect. We developed a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue. Simulated data was used with rotated fiber-bundle masks to create multi-frame stacks to train the model. Super-resolved images are numerically analyzed, which demonstrates that the algorithm can restore images with high quality. The mean structural similarity index measurement (SSIM) improved by a factor of 1.97 compared with linear interpolation. The model was trained using images taken from a single prostate slide, 1343 images were used for training, 336 for validation, and 420 for testing. The model had no prior information about the test images, adding to the robustness of the system. Image reconstruction was completed in 0.03 s for 256 × 256 images indicating future real-time performance is within reach. The combination of fiber bundle rotation and multi-frame image enhancement through machine learning has not been utilized before in an experimental setting but could provide a much-needed improvement to image resolution in practice.
Collapse
Affiliation(s)
- Matthew Eadie
- School of Science and Engineering, Centre for Medical Engineering and Technology, University of Dundee, Dundee DD1 4HN, UK
| | - Jinpeng Liao
- School of Science and Engineering, Centre for Medical Engineering and Technology, University of Dundee, Dundee DD1 4HN, UK
| | - Wael Ageeli
- School of Medicine, Centre for Medical Engineering and Technology, University of Dundee, Dundee DD1 9SY, UK
- Diagnostic Radiology Department, College of Applied Medical Sciences, Jazan University, Al Maarefah Rd, P.O. Box 114, Jazan 45142, Saudi Arabia
| | - Ghulam Nabi
- School of Medicine, Centre for Medical Engineering and Technology, University of Dundee, Dundee DD1 9SY, UK
| | - Nikola Krstajić
- School of Science and Engineering, Centre for Medical Engineering and Technology, University of Dundee, Dundee DD1 4HN, UK
| |
Collapse
|
4
|
Disparity-constrained stereo endoscopic image super-resolution. Int J Comput Assist Radiol Surg 2022; 17:867-875. [PMID: 35377037 DOI: 10.1007/s11548-022-02611-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2022] [Accepted: 03/14/2022] [Indexed: 11/05/2022]
Abstract
PURPOSE With the increasing usage of stereo cameras in computer-assisted surgery techniques, surgeons can benefit from better 3D context of the surgical site in minimally invasive operations. However, since stereo cameras are placed together at the confined endoscope tip, the size of lens and sensors is limited, resulting in low resolution of stereo endoscopic images. How to effectively exploit and utilize stereo information in stereo endoscopic super-resolution (SR) becomes a challenging problem. METHODS In this work, we propose a disparity-constrained stereo super-resolution network (DCSSRnet) to reconstruct images using a stereo image pair. In particular, a disparity constraint mechanism is incorporated into the generation of SR images in the deep neural network framework with effective feature extractors and atrous parallax attention modules. RESULTS Extensive experiments were conducted to evaluate the performance of proposed DCSSRnet on the da Vinci dataset and Medtronic dataset. The results on endoscopic image datasets demonstrate that the proposed approach produces a more effective improvement over current SR methods on both quantitative measurements. The ablation studies further verify the effectiveness of the components of the proposed framework. CONCLUSION The proposed DCSSRnet provides a promising solution on enhancing the spatial resolution of stereo endoscopic image pairs. Specifically, the disparity consistency of the stereo image pair provides informative supervision for image reconstruction. The proposed model can serve as a tool for improving the quality of stereo endoscopic images of endoscopic surgery systems.
Collapse
|
5
|
Tian L, Hunt B, Bell MAL, Yi J, Smith JT, Ochoa M, Intes X, Durr NJ. Deep Learning in Biomedical Optics. Lasers Surg Med 2021; 53:748-775. [PMID: 34015146 PMCID: PMC8273152 DOI: 10.1002/lsm.23414] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Revised: 04/02/2021] [Accepted: 04/15/2021] [Indexed: 01/02/2023]
Abstract
This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.
Collapse
Affiliation(s)
- L. Tian
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, USA
| | - B. Hunt
- Thayer School of Engineering, Dartmouth College, Hanover, NH, USA
| | - M. A. L. Bell
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | - J. Yi
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Ophthalmology, Johns Hopkins University, Baltimore, MD, USA
| | - J. T. Smith
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - M. Ochoa
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - X. Intes
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - N. J. Durr
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
6
|
Szczotka AB, Shakir DI, Clarkson MJ, Pereira SP, Vercauteren T. Zero-Shot Super-Resolution With a Physically-Motivated Downsampling Kernel for Endomicroscopy. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1863-1874. [PMID: 33739921 PMCID: PMC7610492 DOI: 10.1109/tmi.2021.3067512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth high-resolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for ground-truth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.
Collapse
|