1
|
Cao X, Lian Y, Liu Z, Li J, Wang K. Unsupervised spectral reconstruction from RGB images under two lighting conditions. OPTICS LETTERS 2024; 49:1993-1996. [PMID: 38621059 DOI: 10.1364/ol.517007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/27/2023] [Accepted: 03/04/2024] [Indexed: 04/17/2024]
Abstract
Unsupervised spectral reconstruction (SR) aims to recover the hyperspectral image (HSI) from corresponding RGB images without annotations. Existing SR methods achieve it from a single RGB image, hindered by the significant spectral distortion. Although several deep learning-based methods increase the SR accuracy by adding RGB images, their networks are always designed for other image recovery tasks, leaving huge room for improvement. To overcome this problem, we propose a novel, to our knowledge, approach that reconstructs the HSI from a pair of RGB images captured under two illuminations, significantly improving reconstruction accuracy. Specifically, an SR iterative model based on two illuminations is constructed at first. By unfolding the proximal gradient algorithm solving this SR model, an interpretable unsupervised deep network is proposed. All the modules in the proposed network have precise physical meanings, which enable our network to have superior performance and good generalization capability. Experimental results on two public datasets and our real-world images show the proposed method significantly improves both visually and quantitatively as compared with state-of-the-art methods.
Collapse
|
2
|
Ho TT, Tran MT, Cui X, Lin CL, Baek S, Kim WJ, Lee CH, Jin GY, Chae KJ, Choi S. Human-airway surface mesh smoothing based on graph convolutional neural networks. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 246:108061. [PMID: 38341897 DOI: 10.1016/j.cmpb.2024.108061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 01/22/2024] [Accepted: 02/05/2024] [Indexed: 02/13/2024]
Abstract
BACKGROUND AND OBJECTIVE A detailed representation of the airway geometry in the respiratory system is critical for predicting precise airflow and pressure behaviors in computed tomography (CT)-image-based computational fluid dynamics (CFD). The CT-image-based geometry often contains artifacts, noise, and discontinuities due to the so-called stair step effect. Hence, an advanced surface smoothing is necessary. The existing smoothing methods based on the Laplacian operator drastically shrink airway geometries, resulting in the loss of information related to smaller branches. This study aims to introduce an unsupervised airway-mesh-smoothing learning (AMSL) method that preserves the original geometry of the three-dimensional (3D) airway for accurate CT-image-based CFD simulations. METHOD The AMSL method jointly trains two graph convolutional neural networks (GCNNs) defined on airway meshes to filter vertex positions and face normal vectors. In addition, it regularizes a combination of loss functions such as reproducibility, smoothness and consistency of vertex positions, and normal vectors. The AMSL adopts the concept of a deep mesh prior model, and it determines the self-similarity for mesh restoration without using a large dataset for training. Images of the airways of 20 subjects were smoothed by the AMSL method, and among them, the data of two subjects were used for the CFD simulations to assess the effect of airway smoothing on flow properties. RESULTS In 18 of 20 benchmark problems, the proposed smoothing method delivered better results compared with the conventional or state-of-the-art deep learning methods. Unlike the traditional smoothing, the AMSL successfully constructed 20 smoothed airways with airway diameters that were consistent with the original CT images. Besides, CFD simulations with the airways obtained by the AMSL method showed much smaller pressure drop and wall shear stress than the results obtained by the traditional method. CONCLUSIONS The airway model constructed by the AMSL method reproduces branch diameters accurately without any shrinkage, especially in the case of smaller airways. The accurate estimation of airway geometry using a smoothing method is critical for estimating flow properties in CFD simulations.
Collapse
Affiliation(s)
- Thao Thi Ho
- School of Mechanical Engineering, Kyungpook National University, 80 Daehak-ro, Buk-gu, Daegu 41566, South Korea
| | - Minh Tam Tran
- School of Mechanical Engineering, Kyungpook National University, 80 Daehak-ro, Buk-gu, Daegu 41566, South Korea
| | - Xinguang Cui
- School of Aerospace Engineering, Huazhong University of Science and Technology, Wuhan, China
| | - Ching-Long Lin
- Department of Mechanical Engineering, IIHR-Hydroscience and Engineering, University of Iowa, Iowa City, Iowa, USA
| | - Stephen Baek
- School of Data Science, University of Virginia, Charlottesville, VA, USA; Department of Mechanical and Aerospace Engineering, University of Virginia, Charlottesville, VA, USA
| | - Woo Jin Kim
- Department of Internal Medicine and Environmental Health Center, School of Medicine, Kangwon National University Hospital, Kangwon National University, Chuncheon, South Korea
| | - Chang Hyun Lee
- Department of Radiology, College of Medicine, Seoul National University, Seoul National University Hospital, Seoul, South Korea; Department of Radiology, College of Medicine, The University of Iowa, Iowa City, IA, USA
| | - Gong Yong Jin
- Department of Radiology, Research Institute of Clinical Medicine of Jeonbuk National University-Biomedical Research Institute of Jeonbuk National University Hospital, Jeonju, South Korea
| | - Kum Ju Chae
- Department of Radiology, Research Institute of Clinical Medicine of Jeonbuk National University-Biomedical Research Institute of Jeonbuk National University Hospital, Jeonju, South Korea
| | - Sanghun Choi
- School of Mechanical Engineering, Kyungpook National University, 80 Daehak-ro, Buk-gu, Daegu 41566, South Korea.
| |
Collapse
|
3
|
Huang Q, Zhou Y, Liu HF, Bartesaghi A. Joint micrograph denoising and protein localization in cryo-electron microscopy. BIOLOGICAL IMAGING 2024; 4:e4. [PMID: 38571546 PMCID: PMC10988173 DOI: 10.1017/s2633903x24000035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 12/30/2023] [Accepted: 02/05/2024] [Indexed: 04/05/2024]
Abstract
Cryo-electron microscopy (cryo-EM) is an imaging technique that allows the visualization of proteins and macromolecular complexes at near-atomic resolution. The low electron doses used to prevent radiation damage to the biological samples result in images where the power of noise is 100 times stronger than that of the signal. Accurate identification of proteins from these low signal-to-noise ratio (SNR) images is a critical task, as the detected positions serve as inputs for the downstream 3D structure determination process. Current methods either fail to identify all true positives or result in many false positives, especially when analyzing images from smaller-sized proteins that exhibit extremely low contrast, or require manual labeling that can take days to complete. Acknowledging the fact that accurate protein identification is dependent upon the visual interpretability of micrographs, we propose a framework that can perform denoising and detection in a joint manner and enable particle localization under extremely low SNR conditions using self-supervised denoising and particle identification from sparsely annotated data. We validate our approach on three challenging single-particle cryo-EM datasets and projection images from one cryo-electron tomography dataset with extremely low SNR, showing that it outperforms existing state-of-the-art methods used for cryo-EM image analysis by a significant margin. We also evaluate the performance of our algorithm under decreasing SNR conditions and show that our method is more robust to noise than competing methods.
Collapse
Affiliation(s)
- Qinwen Huang
- Department of Computer Science, Duke University, Durham27708, NC, USA
| | - Ye Zhou
- Department of Computer Science, Duke University, Durham27708, NC, USA
| | - Hsuan-Fu Liu
- Department of Biochemistry, Duke University School of Medicine, Durham27705, NC, USA
| | - Alberto Bartesaghi
- Department of Computer Science, Duke University, Durham27708, NC, USA
- Department of Biochemistry, Duke University School of Medicine, Durham27705, NC, USA
- Department of Electrical and Computer Engineering, Duke University, Durham27708, NC, USA
| |
Collapse
|
4
|
Hashimoto F, Onishi Y, Ote K, Tashima H, Reader AJ, Yamaya T. Deep learning-based PET image denoising and reconstruction: a review. Radiol Phys Technol 2024; 17:24-46. [PMID: 38319563 PMCID: PMC10902118 DOI: 10.1007/s12194-024-00780-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Revised: 01/03/2024] [Accepted: 01/04/2024] [Indexed: 02/07/2024]
Abstract
This review focuses on positron emission tomography (PET) imaging algorithms and traces the evolution of PET image reconstruction methods. First, we provide an overview of conventional PET image reconstruction methods from filtered backprojection through to recent iterative PET image reconstruction algorithms, and then review deep learning methods for PET data up to the latest innovations within three main categories. The first category involves post-processing methods for PET image denoising. The second category comprises direct image reconstruction methods that learn mappings from sinograms to the reconstructed images in an end-to-end manner. The third category comprises iterative reconstruction methods that combine conventional iterative image reconstruction with neural-network enhancement. We discuss future perspectives on PET imaging and deep learning technology.
Collapse
Affiliation(s)
- Fumio Hashimoto
- Central Research Laboratory, Hamamatsu Photonics K. K, 5000 Hirakuchi, Hamana-Ku, Hamamatsu, 434-8601, Japan.
- Graduate School of Science and Engineering, Chiba University, 1-33, Yayoicho, Inage-Ku, Chiba, 263-8522, Japan.
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-Ku, Chiba, 263-8555, Japan.
| | - Yuya Onishi
- Central Research Laboratory, Hamamatsu Photonics K. K, 5000 Hirakuchi, Hamana-Ku, Hamamatsu, 434-8601, Japan
| | - Kibo Ote
- Central Research Laboratory, Hamamatsu Photonics K. K, 5000 Hirakuchi, Hamana-Ku, Hamamatsu, 434-8601, Japan
| | - Hideaki Tashima
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-Ku, Chiba, 263-8555, Japan
| | - Andrew J Reader
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, SE1 7EH, UK
| | - Taiga Yamaya
- Graduate School of Science and Engineering, Chiba University, 1-33, Yayoicho, Inage-Ku, Chiba, 263-8522, Japan
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-Ku, Chiba, 263-8555, Japan
| |
Collapse
|
5
|
Feng K, Zeng H, Zhao Y, Kong SG, Bu Y. Unsupervised Spectral Demosaicing With Lightweight Spectral Attention Networks. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:1655-1669. [PMID: 38386587 DOI: 10.1109/tip.2024.3364064] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/24/2024]
Abstract
This paper presents a deep learning-based spectral demosaicing technique trained in an unsupervised manner. Many existing deep learning-based techniques relying on supervised learning with synthetic images, often underperform on real-world images, especially as the number of spectral bands increases. This paper presents a comprehensive unsupervised spectral demosaicing (USD) framework based on the characteristics of spectral mosaic images. This framework encompasses a training method, model structure, transformation strategy, and a well-fitted model selection strategy. To enable the network to dynamically model spectral correlation while maintaining a compact parameter space, we reduce the complexity and parameters of the spectral attention module. This is achieved by dividing the spectral attention tensor into spectral attention matrices in the spatial dimension and spectral attention vector in the channel dimension. This paper also presents Mosaic 25 , a real 25-band hyperspectral mosaic image dataset featuring various objects, illuminations, and materials for benchmarking purposes. Extensive experiments on both synthetic and real-world datasets demonstrate that the proposed method outperforms conventional unsupervised methods in terms of spatial distortion suppression, spectral fidelity, robustness, and computational cost. Our code and dataset are publicly available at https://github.com/polwork/Unsupervised-Spectral-Demosaicing.
Collapse
|
6
|
Wang S, Liu B, Xie F, Chai L. An iterative reconstruction algorithm for unsupervised PET image. Phys Med Biol 2024; 69:055025. [PMID: 38346340 DOI: 10.1088/1361-6560/ad2882] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Accepted: 02/12/2024] [Indexed: 02/28/2024]
Abstract
Objective.In recent years, convolutional neural networks (CNNs) have shown great potential in positron emission tomography (PET) image reconstruction. However, most of them rely on many low-quality and high-quality reference PET image pairs for training, which are not always feasible in clinical practice. On the other hand, many works improve the quality of PET image reconstruction by adding explicit regularization or optimizing the network structure, which may lead to complex optimization problems.Approach.In this paper, we develop a novel iterative reconstruction algorithm by integrating the deep image prior (DIP) framework, which only needs the prior information (e.g. MRI) and sinogram data of patients. To be specific, we construct the objective function as a constrained optimization problem and utilize the existing PET image reconstruction packages to streamline calculations. Moreover, to further improve both the reconstruction quality and speed, we introduce the Nesterov's acceleration part and the restart mechanism in each iteration.Main results.2D experiments on PET data sets based on computer simulations and real patients demonstrate that our proposed algorithm can outperform existing MLEM-GF, KEM and DIPRecon methods.Significance.Unlike traditional CNN methods, the proposed algorithm does not rely on large data sets, but only leverages inter-patient information. Furthermore, we enhance reconstruction performance by optimizing the iterative algorithm. Notably, the proposed method does not require much modification of the basic algorithm, allowing for easy integration into standard implementations.
Collapse
Affiliation(s)
- Siqi Wang
- Engineering Research Center of Metallurgical Automation and Measurement Technology, Wuhan University of Science and Technology, Wuhan 430081, People's Republic of China
| | - Bing Liu
- Engineering Research Center of Metallurgical Automation and Measurement Technology, Wuhan University of Science and Technology, Wuhan 430081, People's Republic of China
| | - Furan Xie
- Engineering Research Center of Metallurgical Automation and Measurement Technology, Wuhan University of Science and Technology, Wuhan 430081, People's Republic of China
| | - Li Chai
- College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, People's Republic of China
| |
Collapse
|
7
|
Yim H, Choe DT, Bae JA, Choi MK, Kang HM, Nguyen KCQ, Ahn S, Bahn SK, Yang H, Hall DH, Kim JS, Lee J. Comparative connectomics of dauer reveals developmental plasticity. Nat Commun 2024; 15:1546. [PMID: 38413604 PMCID: PMC10899629 DOI: 10.1038/s41467-024-45943-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Accepted: 02/06/2024] [Indexed: 02/29/2024] Open
Abstract
A fundamental question in neurodevelopmental biology is how flexibly the nervous system changes during development. To address this, we reconstructed the chemical connectome of dauer, an alternative developmental stage of nematodes with distinct behavioral characteristics, by volumetric reconstruction and automated synapse detection using deep learning. With the basic architecture of the nervous system preserved, structural changes in neurons, large or small, were closely associated with connectivity changes, which in turn evoked dauer-specific behaviors such as nictation. Graph theoretical analyses revealed significant dauer-specific rewiring of sensory neuron connectivity and increased clustering within motor neurons in the dauer connectome. We suggest that the nervous system in the nematode has evolved to respond to harsh environments by developing a quantitatively and qualitatively differentiated connectome.
Collapse
Affiliation(s)
- Hyunsoo Yim
- Department of Biological Sciences, Seoul National University, Seoul, 08826, South Korea
| | - Daniel T Choe
- Department of Biological Sciences, Seoul National University, Seoul, 08826, South Korea
| | - J Alexander Bae
- Research Institute of Basic Sciences, Seoul National University, Seoul, 08826, South Korea
| | - Myung-Kyu Choi
- Research Institute of Basic Sciences, Seoul National University, Seoul, 08826, South Korea
| | - Hae-Mook Kang
- Research Institute of Basic Sciences, Seoul National University, Seoul, 08826, South Korea
| | - Ken C Q Nguyen
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY, 10461, USA
| | - Soungyub Ahn
- Department of Biological Sciences, Seoul National University, Seoul, 08826, South Korea
| | - Sang-Kyu Bahn
- Neural Circuits Research Group, Korea Brain Research Institute, Daegu, 41062, South Korea
- Cognitive Science Research Group, Korea Brain Research Institute, Daegu, 41062, South Korea
| | - Heeseung Yang
- Department of Biological Sciences, Seoul National University, Seoul, 08826, South Korea
| | - David H Hall
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY, 10461, USA
| | - Jinseop S Kim
- Neural Circuits Research Group, Korea Brain Research Institute, Daegu, 41062, South Korea.
- Department of Biological Sciences, Sungkyunkwan University, Suwon-si, Gyeonggi-do, 16419, South Korea.
| | - Junho Lee
- Department of Biological Sciences, Seoul National University, Seoul, 08826, South Korea.
- Research Institute of Basic Sciences, Seoul National University, Seoul, 08826, South Korea.
| |
Collapse
|
8
|
Artesani A, Bruno A, Gelardi F, Chiti A. Empowering PET: harnessing deep learning for improved clinical insight. Eur Radiol Exp 2024; 8:17. [PMID: 38321340 PMCID: PMC10847083 DOI: 10.1186/s41747-023-00413-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Accepted: 11/20/2023] [Indexed: 02/08/2024] Open
Abstract
This review aims to take a journey into the transformative impact of artificial intelligence (AI) on positron emission tomography (PET) imaging. To this scope, a broad overview of AI applications in the field of nuclear medicine and a thorough exploration of deep learning (DL) implementations in cancer diagnosis and therapy through PET imaging will be presented. We firstly describe the behind-the-scenes use of AI for image generation, including acquisition (event positioning, noise reduction though time-of-flight estimation and scatter correction), reconstruction (data-driven and model-driven approaches), restoration (supervised and unsupervised methods), and motion correction. Thereafter, we outline the integration of AI into clinical practice through the applications to segmentation, detection and classification, quantification, treatment planning, dosimetry, and radiomics/radiogenomics combined to tumour biological characteristics. Thus, this review seeks to showcase the overarching transformation of the field, ultimately leading to tangible improvements in patient treatment and response assessment. Finally, limitations and ethical considerations of the AI application to PET imaging and future directions of multimodal data mining in this discipline will be briefly discussed, including pressing challenges to the adoption of AI in molecular imaging such as the access to and interoperability of huge amount of data as well as the "black-box" problem, contributing to the ongoing dialogue on the transformative potential of AI in nuclear medicine.Relevance statementAI is rapidly revolutionising the world of medicine, including the fields of radiology and nuclear medicine. In the near future, AI will be used to support healthcare professionals. These advances will lead to improvements in diagnosis, in the assessment of response to treatment, in clinical decision making and in patient management.Key points• Applying AI has the potential to enhance the entire PET imaging pipeline.• AI may support several clinical tasks in both PET diagnosis and prognosis.• Interpreting the relationships between imaging and multiomics data will heavily rely on AI.
Collapse
Affiliation(s)
- Alessia Artesani
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Milan, Pieve Emanuele, 20090, Italy
| | - Alessandro Bruno
- Department of Business, Law, Economics and Consumer Behaviour "Carlo A. Ricciardi", IULM Libera Università Di Lingue E Comunicazione, Via P. Filargo 38, Milan, 20143, Italy
| | - Fabrizia Gelardi
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Milan, Pieve Emanuele, 20090, Italy.
- Vita-Salute San Raffaele University, Via Olgettina 58, Milan, 20132, Italy.
| | - Arturo Chiti
- Vita-Salute San Raffaele University, Via Olgettina 58, Milan, 20132, Italy
- Department of Nuclear Medicine, IRCCS Ospedale San Raffaele, Via Olgettina 60, Milan, 20132, Italy
| |
Collapse
|
9
|
Balaji V, Song TA, Malekzadeh M, Heidari P, Dutta J. Artificial Intelligence for PET and SPECT Image Enhancement. J Nucl Med 2024; 65:4-12. [PMID: 37945384 PMCID: PMC10755520 DOI: 10.2967/jnumed.122.265000] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 10/10/2023] [Indexed: 11/12/2023] Open
Abstract
Nuclear medicine imaging modalities such as PET and SPECT are confounded by high noise levels and low spatial resolution, necessitating postreconstruction image enhancement to improve their quality and quantitative accuracy. Artificial intelligence (AI) models such as convolutional neural networks, U-Nets, and generative adversarial networks have shown promising outcomes in enhancing PET and SPECT images. This review article presents a comprehensive survey of state-of-the-art AI methods for PET and SPECT image enhancement and seeks to identify emerging trends in this field. We focus on recent breakthroughs in AI-based PET and SPECT image denoising and deblurring. Supervised deep-learning models have shown great potential in reducing radiotracer dose and scan times without sacrificing image quality and diagnostic accuracy. However, the clinical utility of these methods is often limited by their need for paired clean and corrupt datasets for training. This has motivated research into unsupervised alternatives that can overcome this limitation by relying on only corrupt inputs or unpaired datasets to train models. This review highlights recently published supervised and unsupervised efforts toward AI-based PET and SPECT image enhancement. We discuss cross-scanner and cross-protocol training efforts, which can greatly enhance the clinical translatability of AI-based image enhancement tools. We also aim to address the looming question of whether the improvements in image quality generated by AI models lead to actual clinical benefit. To this end, we discuss works that have focused on task-specific objective clinical evaluation of AI models for image enhancement or incorporated clinical metrics into their loss functions to guide the image generation process. Finally, we discuss emerging research directions, which include the exploration of novel training paradigms, curation of larger task-specific datasets, and objective clinical evaluation that will enable the realization of the full translation potential of these models in the future.
Collapse
Affiliation(s)
- Vibha Balaji
- Department of Biomedical Engineering, University of Massachusetts Amherst, Amherst, Massachusetts; and
| | - Tzu-An Song
- Department of Biomedical Engineering, University of Massachusetts Amherst, Amherst, Massachusetts; and
| | - Masoud Malekzadeh
- Department of Biomedical Engineering, University of Massachusetts Amherst, Amherst, Massachusetts; and
| | - Pedram Heidari
- Division of Nuclear Medicine and Molecular Imaging, Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts
| | - Joyita Dutta
- Department of Biomedical Engineering, University of Massachusetts Amherst, Amherst, Massachusetts; and
| |
Collapse
|
10
|
Seifert J, Shao Y, Mosk AP. Noise-robust latent vector reconstruction in ptychography using deep generative models. OPTICS EXPRESS 2024; 32:1020-1033. [PMID: 38175108 DOI: 10.1364/oe.513556] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 12/16/2023] [Indexed: 01/05/2024]
Abstract
Computational imaging is increasingly vital for a broad spectrum of applications, ranging from biological to material sciences. This includes applications where the object is known and sufficiently sparse, allowing it to be described with a reduced number of parameters. When no explicit parameterization is available, a deep generative model can be trained to represent an object in a low-dimensional latent space. In this paper, we harness this dimensionality reduction capability of autoencoders to search for the object solution within the latent space rather than the object space. We demonstrate what we believe to be a novel approach to ptychographic image reconstruction by integrating a deep generative model obtained from a pre-trained autoencoder within an automatic differentiation ptychography (ADP) framework. This approach enables the retrieval of objects from highly ill-posed diffraction patterns, offering an effective method for noise-robust latent vector reconstruction in ptychography. Moreover, the mapping into a low-dimensional latent space allows us to visualize the optimization landscape, which provides insight into the convexity and convergence behavior of the inverse problem. With this work, we aim to facilitate new applications for sparse computational imaging such as when low radiation doses or rapid reconstructions are essential.
Collapse
|
11
|
Zhong Y, Pei Y, Nie K, Zhang Y, Xu T, Zha H. Bi-Graph Reasoning for Masticatory Muscle Segmentation From Cone-Beam Computed Tomography. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3690-3701. [PMID: 37566502 DOI: 10.1109/tmi.2023.3304557] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/13/2023]
Abstract
Automated segmentation of masticatory muscles is a challenging task considering ambiguous soft tissue attachments and image artifacts of low-radiation cone-beam computed tomography (CBCT) images. In this paper, we propose a bi-graph reasoning model (BGR) for the simultaneous detection and segmentation of multi-category masticatory muscles from CBCTs. The BGR exploits the local and long-range interdependencies of regions of interest and category-specific prior knowledge of masticatory muscles by reasoning on the category graph and the region graph. The category graph of the learnable muscle prior knowledge handles high-level dependencies of muscle categories, enhancing the feature representation with noise-agnostic category knowledge. The region graph models both local and global dependencies of the candidate muscle regions of interest. The proposed BGR accommodates the high-level dependencies and enhances the region features in the presence of entangled soft tissue and image artifacts. We evaluated the proposed approach by segmenting masticatory muscles on clinically acquired CBCTs. Extensive experimental results show that the BGR effectively segments masticatory muscles with state-of-the-art accuracy.
Collapse
|
12
|
Li Y, Yang J, Yu T, Chi J, Liu F. Global attention-enabled texture enhancement network for MR image reconstruction. Magn Reson Med 2023; 90:1919-1931. [PMID: 37382206 DOI: 10.1002/mrm.29785] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Revised: 05/23/2023] [Accepted: 06/14/2023] [Indexed: 06/30/2023]
Abstract
PURPOSE Although recent convolutional neural network (CNN) methodologies have shown promising results in fast MR imaging, there is still a desire to explore how they can be used to learn the frequency characteristics of multicontrast images and reconstruct texture details. METHODS A global attention-enabled texture enhancement network (GATE-Net) with a frequency-dependent feature extraction module (FDFEM) and convolution-based global attention module (GAM) is proposed to address the highly under-sampling MR image reconstruction problem. First, FDFEM enables GATE-Net to effectively extract high-frequency features from shareable information of multicontrast images to improve the texture details of reconstructed images. Second, GAM with less computation complexity has the receptive field of the entire image, which can fully explore useful shareable information of multi-contrast images and suppress less beneficial shareable information. RESULTS The ablation studies are conducted to evaluate the effectiveness of the proposed FDFEM and GAM. Experimental results under various acceleration rates and datasets consistently demonstrate the superiority of GATE-Net, in terms of peak signal-to-noise ratio, structural similarity and normalized mean square error. CONCLUSION A global attention-enabled texture enhancement network is proposed. it can be applied to multicontrast MR image reconstruction tasks with different acceleration rates and datasets and achieves superior performance in comparison with state-of-the-art methods.
Collapse
Affiliation(s)
- Yingnan Li
- College of Electronics and Information, Qingdao University, Qingdao, Shandong, China
| | - Jie Yang
- College of Mechanical and Electrical Engineering, Qingdao University, Qingdao, Shandong, China
| | - Teng Yu
- College of Electronics and Information, Qingdao University, Qingdao, Shandong, China
| | - Jieru Chi
- College of Electronics and Information, Qingdao University, Qingdao, Shandong, China
| | - Feng Liu
- School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, Queensland, Australia
| |
Collapse
|
13
|
Shinohara H, Hori K, Hashimoto T. Deep learning study on the mechanism of edge artifacts in point spread function reconstruction for numerical brain images. Ann Nucl Med 2023; 37:596-604. [PMID: 37610591 DOI: 10.1007/s12149-023-01862-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Accepted: 08/07/2023] [Indexed: 08/24/2023]
Abstract
OBJECTIVE Non-blinded image deblurring with deep learning was performed on blurred numerical brain images without point spread function (PSF) reconstruction to obtain edge artifacts (EA)-free images. This study uses numerical simulation to investigate the mechanism of EA in PSF reconstruction based on the spatial frequency characteristics of EA-free images. METHODS In 256 × 256 matrix brain images, the signal values of gray matter (GM), white matter, and cerebrospinal fluid were set to 1, 0.25, and 0.05, respectively. We assumed ideal projection data of a two-dimensional (2D) parallel beam with no degradation factors other than detector response blur to precisely grasp EA using the PSF reconstruction algorithm from blurred projection data. The detector response was assumed to be a shift-invariant and one-dimensional (1D) Gaussian function with 2-5 mm full width at half maximum (FWHM). Images without PSF reconstruction (non-PSF), PSF reconstruction without regularization (PSF) and with regularization of relative difference function (PSF-RD) were generated by ordered subset expectation maximization (OSEM). For non-PSF, the image deblurring with a deep image prior (DIP) was applied using a 2D Gaussian function with 2-5 mm FWHM. The 1D object-specific modulation transfer function (1D-OMTF), which is the ratio of 1D amplitude spectrum of the original and reconstructed images, was used as the index of spatial frequency characteristics. RESULTS When the detector response was greater than 3 mm FWHM, EA in PSF was observed in GM borders and narrow GM. No remarkable EA was observed in the DIP, and the FWHM estimated from the recovery coefficient for the deblurred image of non-PSF at 5 mm FWHM was reduced to 3 mm or less. PSF of 5 mm FWHM showed higher spatial frequency characteristics than that of DIP up to around 2.2 cycles/cm but was lower than the latter after 3 cycles/cm. PSF-RD showed almost the same spatial frequency characteristics as that of DIP above 3 cycles/cm but was inferior below 3 cycles/cm. PSF-RD has a lower spatial resolution than DIP. CONCLUSIONS Unlike DIP, PSF lacks high-frequency components around the Nyquist frequency, generating EA. PSF-RD mitigates EA while simultaneously suppressing the signal, diminishing spatial resolution.
Collapse
Affiliation(s)
- Hiroyuki Shinohara
- Faculty of Health Sciences, Tokyo Metropolitan University, 7-2-10, Higasi-ogu, Arakawa-ku, Tokyo, 116-8551, Japan.
- Department of Radiology, Showa University Fujigaoka Hospital, 1-30, Fujigaoka, Yokohama-shi, 227-8501, Japan.
| | - Kensuke Hori
- Department of Radiological Technology, Faculty of Health Science, Juntendo University, 1-5-32, Yushima, Bunkyo-ku, Tokyo, 113-0034, Japan
| | - Takeyuki Hashimoto
- Department of Radiological Technology, Faculty of Health Science, Kyorin University, 5-4-1 Shimorenjaku, Mitaka-shi, Tokyo, 181-8612, Japan
| |
Collapse
|
14
|
Diao W, Zhang F, Sun J, Xing Y, Zhang K, Bruzzone L. ZeRGAN: Zero-Reference GAN for Fusion of Multispectral and Panchromatic Images. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:8195-8209. [PMID: 34982704 DOI: 10.1109/tnnls.2021.3137373] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
In this article, we present a new pansharpening method, a zero-reference generative adversarial network (ZeRGAN), which fuses low spatial resolution multispectral (LR MS) and high spatial resolution panchromatic (PAN) images. In the proposed method, zero-reference indicates that it does not require paired reduced-scale images or unpaired full-scale images for training. To obtain accurate fusion results, we establish an adversarial game between a set of multiscale generators and their corresponding discriminators. Through multiscale generators, the fused high spatial resolution MS (HR MS) images are progressively produced from LR MS and PAN images, while the discriminators aim to distinguish the differences of spatial information between the HR MS images and the PAN images. In other words, the HR MS images are generated from LR MS and PAN images after the optimization of ZeRGAN. Furthermore, we construct a nonreference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss. Through the minimization of the total loss, the spatial details in the HR MS images can be enhanced efficiently. Extensive experiments are implemented on datasets acquired by different satellites. The results demonstrate that the effectiveness of the proposed method compared with the state-of-the-art methods. The source code is publicly available at https://github.com/RSMagneto/ZeRGAN.
Collapse
|
15
|
Jung K, Mandija S, Cui C, Kim J, Al‐masni MA, Meerbothe TG, Park M, van den Berg CAT, Kim D. Data-driven electrical conductivity brain imaging using 3 T MRI. Hum Brain Mapp 2023; 44:4986-5001. [PMID: 37466309 PMCID: PMC10502651 DOI: 10.1002/hbm.26421] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 06/14/2023] [Accepted: 07/03/2023] [Indexed: 07/20/2023] Open
Abstract
Magnetic resonance electrical properties tomography (MR-EPT) is a non-invasive measurement technique that derives the electrical properties (EPs, e.g., conductivity or permittivity) of tissues in the radiofrequency range (64 MHz for 1.5 T and 128 MHz for 3 T MR systems). Clinical studies have shown the potential of tissue conductivity as a biomarker. To date, model-based conductivity reconstructions rely on numerical assumptions and approximations, leading to inaccuracies in the reconstructed maps. To address such limitations, we propose an artificial neural network (ANN)-based non-linear conductivity estimator trained on simulated data for conductivity brain imaging. Network training was performed on 201 synthesized T2-weighted spin-echo (SE) data obtained from the finite-difference time-domain (FDTD) electromagnetic (EM) simulation. The dataset was composed of an approximated T2-w SE magnitude and transceive phase information. The proposed method was tested three in-silico and in-vivo on two volunteers and three patients' data. For comparison purposes, various conventional phase-based EPT reconstruction methods were used that ignoreB 1 + magnitude information, such as Savitzky-Golay kernel combined with Gaussian filter (S-G Kernel), phase-based convection-reaction EPT (cr-EPT), magnitude-weighted polynomial-fitting phase-based EPT (Poly-Fit), and integral-based phase-based EPT (Integral-based). From the in-silico experiments, quantitative analysis showed that the proposed method provides more accurate and improved quality (e.g., high structural preservation) conductivity maps compared to conventional reconstruction methods. Representatively, in the healthy brain in-silico phantom experiment, the proposed method yielded mean conductivity values of 1.97 ± 0.20 S/m for CSF, 0.33 ± 0.04 S/m for WM, and 0.52 ± 0.08 S/m for GM, which were closer to the ground-truth conductivity (2.00, 0.30, 0.50 S/m) than the integral-based method (2.56 ± 2.31, 0.39 ± 0.12, 0.68 ± 0.33 S/m). In-vivo ANN-based conductivity reconstructions were also of improved quality compared to conventional reconstructions and demonstrated network generalizability and robustness to in-vivo data and pathologies. The reported in-vivo brain conductivity values were in agreement with literatures. In addition, the proposed method was observed for various SNR levels (SNR levels = 10, 20, 40, and 58) and repeatability conditions (the eight acquisitions with the number of signal averages = 1). The preliminary investigations on brain tumor patient datasets suggest that the network trained on simulated dataset can generalize to unforeseen in-vivo pathologies, thus demonstrating its potential for clinical applications.
Collapse
Affiliation(s)
- Kyu‐Jin Jung
- Department of Electrical and Electronic EngineeringYonsei UniversitySeoulRepublic of Korea
| | - Stefano Mandija
- Computational Imaging Group for MR Therapy and DiagnosticsUniversity Medical Center UtrechtUtrechtThe Netherlands
- Department of RadiotherapyUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Chuanjiang Cui
- Department of Electrical and Electronic EngineeringYonsei UniversitySeoulRepublic of Korea
| | - Jun‐Hyeong Kim
- Department of Electrical and Electronic EngineeringYonsei UniversitySeoulRepublic of Korea
| | - Mohammed A. Al‐masni
- Department of Artificial IntelligenceCollege of Software & Convergence Technology, Daeyang AI Center, Sejong UniversitySeoulRepublic of Korea
| | - Thierry G. Meerbothe
- Computational Imaging Group for MR Therapy and DiagnosticsUniversity Medical Center UtrechtUtrechtThe Netherlands
- Department of RadiotherapyUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Mina Park
- Department of Radiology, Gangnam Severance HospitalYonsei University College of MedicineSeoulRepublic of Korea
| | - Cornelis A. T. van den Berg
- Computational Imaging Group for MR Therapy and DiagnosticsUniversity Medical Center UtrechtUtrechtThe Netherlands
- Department of RadiotherapyUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Dong‐Hyun Kim
- Department of Electrical and Electronic EngineeringYonsei UniversitySeoulRepublic of Korea
| |
Collapse
|
16
|
Akindele RG, Yu M, Kanda PS, Owoola EO, Aribilola I. Denoising of Nifti (MRI) Images with a Regularized Neighborhood Pixel Similarity Wavelet Algorithm. SENSORS (BASEL, SWITZERLAND) 2023; 23:7780. [PMID: 37765837 PMCID: PMC10536345 DOI: 10.3390/s23187780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/24/2023] [Revised: 07/26/2023] [Accepted: 09/04/2023] [Indexed: 09/29/2023]
Abstract
The recovery of semantics from corrupted images is a significant challenge in image processing. Noise can obscure features, interfere with accurate analysis, and bias results. To address this issue, the Regularized Neighborhood Pixel Similarity Wavelet algorithm (PixSimWave) was developed for denoising Nifti (magnetic resonance imaging (MRI)). The PixSimWave algorithm uses regularized pixel similarity detection to improve the accuracy of noise reduction by creating patches to analyze the intensity of pixels and locate matching pixels, as well as adaptive neighborhood filtering to estimate noisy pixel values by allocating each pixel a weight based on its similarity. The wavelet transform breaks down the image into scales and orientations, allowing a sparse image representation to allocate a soft threshold on its similarity to the original pixels. The proposed method was evaluated on simulated and raw T1w MRIs, outperforming other methods in terms of an SSIM value of 0.9908 for a low Rician noise level of 3% and 0.9881 for a high noise level of 17%. The addition of Gaussian noise improved PSNR and SSIM, with the results indicating that the proposed method outperformed other models while preserving edges and textures. In summary, the PixSimWave algorithm is a viable noise-elimination approach that employs both sparse wavelet coefficients and regularized similarity with decreased computation time, improving the accuracy of noise reduction in images.
Collapse
Affiliation(s)
- Romoke Grace Akindele
- School of Electronics and Information Engineering, Hebei University of Technology, Tianjin 300401, China; (P.S.K.); (E.O.O.)
| | - Ming Yu
- School of Electronics and Information Engineering, Hebei University of Technology, Tianjin 300401, China; (P.S.K.); (E.O.O.)
- School of Artificial Intelligence, Hebei University of Technology, Tianjin 300401, China
| | - Paul Shekonya Kanda
- School of Electronics and Information Engineering, Hebei University of Technology, Tianjin 300401, China; (P.S.K.); (E.O.O.)
| | - Eunice Oluwabunmi Owoola
- School of Electronics and Information Engineering, Hebei University of Technology, Tianjin 300401, China; (P.S.K.); (E.O.O.)
| | - Ifeoluwapo Aribilola
- Software Research Institute, Technological University of the Shannon, Midlands Midwest, Co. Westmeath, N37 HD68 Athlone, Ireland;
| |
Collapse
|
17
|
Muller FM, Maebe J, Vanhove C, Vandenberghe S. Dose reduction and image enhancement in micro-CT using deep learning. Med Phys 2023; 50:5643-5656. [PMID: 36994779 DOI: 10.1002/mp.16385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Revised: 02/14/2023] [Accepted: 03/09/2023] [Indexed: 03/31/2023] Open
Abstract
BACKGROUND In preclinical settings, micro-computed tomography (CT) provides a powerful tool to acquire high resolution anatomical images of rodents and offers the advantage to in vivo non-invasively assess disease progression and therapy efficacy. Much higher resolutions are needed to achieve scale-equivalent discriminatory capabilities in rodents as those in humans. High resolution imaging however comes at the expense of increased scan times and higher doses. Specifically, with preclinical longitudinal imaging, there are concerns that dose accumulation may affect experimental outcomes of animal models. PURPOSE Dose reduction efforts under the ALARA (as low as reasonably achievable) principles are thus a key point of attention. However, low dose CT acquisitions inherently induce higher noise levels which deteriorate image quality and negatively impact diagnostic performance. Many denoising techniques already exist, and deep learning (DL) has become increasingly popular for image denoising, but research has mostly focused on clinical CT with limited studies conducted on preclinical CT imaging. We investigate the potential of convolutional neural networks (CNN) for restoring high quality micro-CT images from low dose (noisy) images. The novelty of the CNN denoising frameworks presented in this work consists of utilizing image pairs with realistic CT noise present in the input as well as the target image used for the model training; a noisier image acquired with a low dose protocol is matched to a less noisy image acquired with a higher dose scan of the same mouse. METHODS Low and high dose ex vivo micro-CT scans of 38 mice were acquired. Two CNN models, based on a 2D and 3D four-layer U-Net, were trained with mean absolute error (30 training, 4 validation and 4 test sets). To assess denoising performance, ex vivo mice and phantom data were used. Both CNN approaches were compared to existing methods, like spatial filtering (Gaussian, Median, Wiener) and iterative total variation image reconstruction algorithm. Image quality metrics were derived from the phantom images. A first observer study (n = 23) was set-up to rank overall quality of differently denoised images. A second observer study (n = 18) estimated the dose reduction factor of the investigated 2D CNN method. RESULTS Visual and quantitative results show that both CNN algorithms exhibit superior performance in terms of noise suppression, structural preservation and contrast enhancement over comparator methods. The quality scoring by 23 medical imaging experts also indicates that the investigated 2D CNN approach is consistently evaluated as the best performing denoising method. Results from the second observer study and quantitative measurements suggest that CNN-based denoising could offer a 2-4× dose reduction, with an estimated dose reduction factor of about 3.2 for the considered 2D network. CONCLUSIONS Our results demonstrate the potential of DL in micro-CT for higher quality imaging at low dose acquisition settings. In the context of preclinical research, this offers promising future prospects for managing the cumulative severity effects of radiation in longitudinal studies.
Collapse
Affiliation(s)
- Florence M Muller
- Medical Image and Signal Processing (MEDISIP), Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, Ghent, Belgium
| | - Jens Maebe
- Medical Image and Signal Processing (MEDISIP), Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, Ghent, Belgium
| | - Christian Vanhove
- Medical Image and Signal Processing (MEDISIP), Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, Ghent, Belgium
| | - Stefaan Vandenberghe
- Medical Image and Signal Processing (MEDISIP), Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, Ghent, Belgium
| |
Collapse
|
18
|
Bouchama L, Dorizzi B, Klossa J, Gottesman Y. A Physics-Inspired Deep Learning Framework for an Efficient Fourier Ptychographic Microscopy Reconstruction under Low Overlap Conditions. SENSORS (BASEL, SWITZERLAND) 2023; 23:6829. [PMID: 37571611 PMCID: PMC10422347 DOI: 10.3390/s23156829] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Revised: 07/20/2023] [Accepted: 07/28/2023] [Indexed: 08/13/2023]
Abstract
Two-dimensional observation of biological samples at hundreds of nanometers resolution or even below is of high interest for many sensitive medical applications. Recent advances have been obtained over the last ten years with computational imaging. Among them, Fourier Ptychographic Microscopy is of particular interest because of its important super-resolution factor. In complement to traditional intensity images, phase images are also produced. A large set of N raw images (with typically N = 225) is, however, required because of the reconstruction process that is involved. In this paper, we address the problem of FPM image reconstruction using a few raw images only (here, N = 37) as is highly desirable to increase microscope throughput. In contrast to previous approaches, we develop an algorithmic approach based on a physics-informed optimization deep neural network and statistical reconstruction learning. We demonstrate its efficiency with the help of simulations. The forward microscope image formation model is explicitly introduced in the deep neural network model to optimize its weights starting from an initialization that is based on statistical learning. The simulation results that are presented demonstrate the conceptual benefits of the approach. We show that high-quality images are effectively reconstructed without any appreciable resolution degradation. The learning step is also shown to be mandatory.
Collapse
Affiliation(s)
- Lyes Bouchama
- Samovar, Télécom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France; (B.D.); (Y.G.)
- TRIBVN/T-Life, 92800 Puteaux, France;
| | - Bernadette Dorizzi
- Samovar, Télécom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France; (B.D.); (Y.G.)
| | | | - Yaneck Gottesman
- Samovar, Télécom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France; (B.D.); (Y.G.)
| |
Collapse
|
19
|
Millard C, Chiew M. A Theoretical Framework for Self-Supervised MR Image Reconstruction Using Sub-Sampling via Variable Density Noisier2Noise. IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING 2023; 9:707-720. [PMID: 37600280 PMCID: PMC7614963 DOI: 10.1109/tci.2023.3299212] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/22/2023]
Abstract
In recent years, there has been attention on leveraging the statistical modeling capabilities of neural networks for reconstructing sub-sampled Magnetic Resonance Imaging (MRI) data. Most proposed methods assume the existence of a representative fully-sampled dataset and use fully-supervised training. However, for many applications, fully sampled training data is not available, and may be highly impractical to acquire. The development and understanding of self-supervised methods, which use only sub-sampled data for training, are therefore highly desirable. This work extends the Noisier2Noise framework, which was originally constructed for self-supervised denoising tasks, to variable density sub-sampled MRI data. We use the Noisier2Noise framework to analytically explain the performance of Self-Supervised Learning via Data Undersampling (SSDU), a recently proposed method that performs well in practice but until now lacked theoretical justification. Further, we propose two modifications of SSDU that arise as a consequence of the theoretical developments. Firstly, we propose partitioning the sampling set so that the subsets have the same type of distribution as the original sampling mask. Secondly, we propose a loss weighting that compensates for the sampling and partitioning densities. On the fastMRI dataset we show that these changes significantly improve SSDU's image restoration quality and robustness to the partitioning parameters.
Collapse
Affiliation(s)
- Charles Millard
- the Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU Oxford, U.K
| | - Mark Chiew
- the Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU Oxford, U.K., and with the Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada, and also with the Canada and Physical Sciences, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| |
Collapse
|
20
|
Hashimoto F, Onishi Y, Ote K, Tashima H, Yamaya T. Fully 3D implementation of the end-to-end deep image prior-based PET image reconstruction using block iterative algorithm. Phys Med Biol 2023; 68:155009. [PMID: 37406637 DOI: 10.1088/1361-6560/ace49c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Accepted: 07/05/2023] [Indexed: 07/07/2023]
Abstract
Objective. Deep image prior (DIP) has recently attracted attention owing to its unsupervised positron emission tomography (PET) image reconstruction method, which does not require any prior training dataset. In this paper, we present the first attempt to implement an end-to-end DIP-based fully 3D PET image reconstruction method that incorporates a forward-projection model into a loss function.Approach. A practical implementation of a fully 3D PET image reconstruction could not be performed at present because of a graphics processing unit memory limitation. Consequently, we modify the DIP optimization to a block iteration and sequential learning of an ordered sequence of block sinograms. Furthermore, the relative difference penalty (RDP) term is added to the loss function to enhance the quantitative accuracy of the PET image.Main results. We evaluated our proposed method using Monte Carlo simulation with [18F]FDG PET data of a human brain and a preclinical study on monkey-brain [18F]FDG PET data. The proposed method was compared with the maximum-likelihood expectation maximization (EM), maximuma posterioriEM with RDP, and hybrid DIP-based PET reconstruction methods. The simulation results showed that, compared with other algorithms, the proposed method improved the PET image quality by reducing statistical noise and better preserved the contrast of brain structures and inserted tumors. In the preclinical experiment, finer structures and better contrast recovery were obtained with the proposed method.Significance.The results indicated that the proposed method could produce high-quality images without a prior training dataset. Thus, the proposed method could be a key enabling technology for the straightforward and practical implementation of end-to-end DIP-based fully 3D PET image reconstruction.
Collapse
Affiliation(s)
- Fumio Hashimoto
- Central Research Laboratory, Hamamatsu Photonics K. K., 5000 Hirakuchi, Hamakita-ku, Hamamatsu 434-8601, Japan
- Graduate School of Science and Engineering, Chiba University, 1-33, Yayoicho, Inage-ku, Chiba, 263-8522, Japan
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-ku, Chiba, 263-8555, Japan
| | - Yuya Onishi
- Central Research Laboratory, Hamamatsu Photonics K. K., 5000 Hirakuchi, Hamakita-ku, Hamamatsu 434-8601, Japan
| | - Kibo Ote
- Central Research Laboratory, Hamamatsu Photonics K. K., 5000 Hirakuchi, Hamakita-ku, Hamamatsu 434-8601, Japan
| | - Hideaki Tashima
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-ku, Chiba, 263-8555, Japan
| | - Taiga Yamaya
- Graduate School of Science and Engineering, Chiba University, 1-33, Yayoicho, Inage-ku, Chiba, 263-8522, Japan
- National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-ku, Chiba, 263-8555, Japan
| |
Collapse
|
21
|
Seong B, Kim I, Moon T, Ranathunga M, Kim D, Joo C. Untrained deep learning-based differential phase-contrast microscopy. OPTICS LETTERS 2023; 48:3607-3610. [PMID: 37390192 DOI: 10.1364/ol.493391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Accepted: 05/31/2023] [Indexed: 07/02/2023]
Abstract
Quantitative differential phase-contrast (DPC) microscopy produces phase images of transparent objects based on a number of intensity images. To reconstruct the phase, in DPC microscopy, a linearized model for weakly scattering objects is considered; this limits the range of objects to be imaged, and requires additional measurements and complicated algorithms to correct for system aberrations. Here, we present a self-calibrated DPC microscope using an untrained neural network (UNN), which incorporates the nonlinear image formation model. Our method alleviates the restrictions on the object to be imaged and simultaneously reconstructs the complex object information and aberrations, without any training dataset. We demonstrate the viability of UNN-DPC microscopy through both numerical simulations and LED microscope-based experiments.
Collapse
|
22
|
Ullmann D, Taran O, Voloshynovskiy S. Multivariate Time Series Information Bottleneck. ENTROPY (BASEL, SWITZERLAND) 2023; 25:e25050831. [PMID: 37238586 DOI: 10.3390/e25050831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/22/2023] [Revised: 05/10/2023] [Accepted: 05/17/2023] [Indexed: 05/28/2023]
Abstract
Time series (TS) and multiple time series (MTS) predictions have historically paved the way for distinct families of deep learning models. The temporal dimension, distinguished by its evolutionary sequential aspect, is usually modeled by decomposition into the trio of "trend, seasonality, noise", by attempts to copy the functioning of human synapses, and more recently, by transformer models with self-attention on the temporal dimension. These models may find applications in finance and e-commerce, where any increase in performance of less than 1% has large monetary repercussions, they also have potential applications in natural language processing (NLP), medicine, and physics. To the best of our knowledge, the information bottleneck (IB) framework has not received significant attention in the context of TS or MTS analyses. One can demonstrate that a compression of the temporal dimension is key in the context of MTS. We propose a new approach with partial convolution, where a time sequence is encoded into a two-dimensional representation resembling images. Accordingly, we use the recent advances made in image extension to predict an unseen part of an image from a given one. We show that our model compares well with traditional TS models, has information-theoretical foundations, and can be easily extended to more dimensions than only time and space. An evaluation of our multiple time series-information bottleneck (MTS-IB) model proves its efficiency in electricity production, road traffic, and astronomical data representing solar activity, as recorded by NASA's interface region imaging spectrograph (IRIS) satellite.
Collapse
Affiliation(s)
- Denis Ullmann
- Faculty of Science, University of Geneva, CUI, 1227 Carouge, Switzerland
| | - Olga Taran
- Faculty of Science, University of Geneva, CUI, 1227 Carouge, Switzerland
| | | |
Collapse
|
23
|
Tian H, Zhao J, Zhu J, Tang X, Cui G, Hou C. Dual-camera compressive hyperspectral imaging based on deep image prior and a guided filter. APPLIED OPTICS 2023; 62:3649-3659. [PMID: 37706982 DOI: 10.1364/ao.483993] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Accepted: 04/11/2023] [Indexed: 09/15/2023]
Abstract
Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.
Collapse
|
24
|
Han S, Remedios SW, Schär M, Carass A, Prince JL. ESPRESO: An algorithm to estimate the slice profile of a single magnetic resonance image. Magn Reson Imaging 2023; 98:155-163. [PMID: 36702167 DOI: 10.1016/j.mri.2023.01.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Accepted: 01/14/2023] [Indexed: 01/25/2023]
Abstract
To reduce scan time, magnetic resonance (MR) images are often acquired using 2D multi-slice protocols with thick slices that may also have gaps between them. The resulting image volumes have lower resolution in the through-plane direction than in the in-plane direction, and the through-plane resolution is in part characterized by the protocol's slice profile which acts as a through-plane point spread function (PSF). Although super-resolution (SR) has been shown to improve the visualization and down-stream processing of 2D multi-slice MR acquisitions, previous algorithms are usually unaware of the true slice profile, which may lead to sub-optimal SR performance. In this work, we present an algorithm to estimate the slice profile of a 2D multi-slice acquisition given only its own image volume without any external training data. We assume that an anatomical image is isotropic in the sense that, after accounting for a correctly estimated slice profile, the image patches along different orientations have the same probability distribution. Our proposed algorithm uses a modified generative adversarial network (GAN) where the generator network estimates the slice profile to reduce the resolution of the in-plane direction, and the discriminator network determines whether a direction is generated or real low resolution. The proposed algorithm, ESPRESO, which stands for "estimating the slice profile for resolution enhancement of a single image only", was tested with a state-of-the-art internally supervised SR algorithm. Specifically, ESPRESO is used to create training data for this SR algorithm, and results show improvements when ESPRESO is used over commonly-used PSFs.
Collapse
Affiliation(s)
- Shuo Han
- The Department of Biomedical Engineering, The Johns Hopkins University, Baltimore 21218, MD, USA.
| | - Samuel W Remedios
- The Department of Computer Science, The Johns Hopkins University, Baltimore 21218, MD, USA.
| | - Michael Schär
- The Department of Radiology, The Johns Hopkins School of Medicine, Baltimore 21205, MD, USA.
| | - Aaron Carass
- The Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore 21218, MD, USA.
| | - Jerry L Prince
- The Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore 21218, MD, USA.
| |
Collapse
|
25
|
Li S, Gong K, Badawi RD, Kim EJ, Qi J, Wang G. Neural KEM: A Kernel Method With Deep Coefficient Prior for PET Image Reconstruction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:785-796. [PMID: 36288234 PMCID: PMC10081957 DOI: 10.1109/tmi.2022.3217543] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Image reconstruction of low-count positron emission tomography (PET) data is challenging. Kernel methods address the challenge by incorporating image prior information in the forward model of iterative PET image reconstruction. The kernelized expectation-maximization (KEM) algorithm has been developed and demonstrated to be effective and easy to implement. A common approach for a further improvement of the kernel method would be adding an explicit regularization, which however leads to a complex optimization problem. In this paper, we propose an implicit regularization for the kernel method by using a deep coefficient prior, which represents the kernel coefficient image in the PET forward model using a convolutional neural-network. To solve the maximum-likelihood neural network-based reconstruction problem, we apply the principle of optimization transfer to derive a neural KEM algorithm. Each iteration of the algorithm consists of two separate steps: a KEM step for image update from the projection data and a deep-learning step in the image domain for updating the kernel coefficient image using the neural network. This optimization algorithm is guaranteed to monotonically increase the data likelihood. The results from computer simulations and real patient data have demonstrated that the neural KEM can outperform existing KEM and deep image prior methods.
Collapse
|
26
|
Lam F, Peng X, Liang ZP. High-Dimensional MR Spatiospectral Imaging by Integrating Physics-Based Modeling and Data-Driven Machine Learning: Current progress and future directions. IEEE SIGNAL PROCESSING MAGAZINE 2023; 40:101-115. [PMID: 37538148 PMCID: PMC10398845 DOI: 10.1109/msp.2022.3203867] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/05/2023]
Abstract
Magnetic resonance spectroscopic imaging (MRSI) offers a unique molecular window into the physiological and pathological processes in the human body. However, the applications of MRSI have been limited by a number of long-standing technical challenges due to high dimensionality and low signal-to-noise ratio (SNR). Recent technological developments integrating physics-based modeling and data-driven machine learning that exploit unique physical and mathematical properties of MRSI signals have demonstrated impressive performance in addressing these challenges for rapid, high-resolution, quantitative MRSI. This paper provides a systematic review of these progresses in the context of MRSI physics and offers perspectives on promising future directions.
Collapse
Affiliation(s)
- Fan Lam
- Department of Bioengineering, University of Illinois Urbana-Champaign, Urbana, IL, 61801 USA
- Beckman Institute for Advanced Science and Technology, Department of Electrical and Computer Engineering and Cancer Center at Illinois, University of Illinois Urbana-Champaign
| | - Xi Peng
- Department of Radiology, Mayo Clinic, Rochester, MN, 55905, USA
| | - Zhi-Pei Liang
- Beckman Institute for Advanced Science and Technology, Department of Electrical and Computer Engineering and Cancer Center at Illinois, University of Illinois Urbana-Champaign
| |
Collapse
|
27
|
Huang Q, Zhou Y, Liu HF, Bartesaghi A. Multiple-image super-resolution of cryo-electron micrographs based on deep internal learning. BIOLOGICAL IMAGING 2023; 3:e3. [PMID: 38510165 PMCID: PMC10951919 DOI: 10.1017/s2633903x2300003x] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 12/27/2022] [Accepted: 01/23/2023] [Indexed: 03/22/2024]
Abstract
Single-particle cryo-electron microscopy (cryo-EM) is a powerful imaging modality capable of visualizing proteins and macromolecular complexes at near-atomic resolution. The low electron-doses used to prevent radiation damage to the biological samples, however, result in images where the power of the noise is 100 times greater than the power of the signal. To overcome these low signal-to-noise ratios (SNRs), hundreds of thousands of particle projections are averaged to determine the three-dimensional structure of the molecule of interest. The sampling requirements of high-resolution imaging impose limitations on the pixel sizes that can be used for acquisition, limiting the size of the field of view and requiring data collection sessions of several days to accumulate sufficient numbers of particles. Meanwhile, recent image super-resolution (SR) techniques based on neural networks have shown state-of-the-art performance on natural images. Building on these advances, here, we present a multiple-image SR algorithm based on deep internal learning designed specifically to work under low-SNR conditions. Our approach leverages the internal image statistics of cryo-EM movies and does not require training on ground-truth data. When applied to single-particle datasets of apoferritin and T20S proteasome, we show that the resolution of the 3D structure obtained from SR micrographs can surpass the limits imposed by the imaging system. Our results indicate that the combination of low magnification imaging with in silico image SR has the potential to accelerate cryo-EM data collection by virtue of including more particles in each exposure and doing so without sacrificing resolution.
Collapse
Affiliation(s)
- Qinwen Huang
- Department of Computer Science, Duke University, Durham, North Carolina, USA
| | - Ye Zhou
- Department of Computer Science, Duke University, Durham, North Carolina, USA
| | - Hsuan-Fu Liu
- Department of Biochemistry, Duke University School of Medicine, Durham, North Carolina, USA
| | - Alberto Bartesaghi
- Department of Computer Science, Duke University, Durham, North Carolina, USA
- Department of Biochemistry, Duke University School of Medicine, Durham, North Carolina, USA
- Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina, USA
| |
Collapse
|
28
|
Zhang W, Hu T, Li Z, Sun Z, Jia K, Dou H, Feng J, Pogue BW. Selfrec-Net: self-supervised deep learning approach for the reconstruction of Cherenkov-excited luminescence scanned tomography. BIOMEDICAL OPTICS EXPRESS 2023; 14:783-798. [PMID: 36874507 PMCID: PMC9979688 DOI: 10.1364/boe.480429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Revised: 12/23/2022] [Accepted: 01/04/2023] [Indexed: 06/18/2023]
Abstract
As an emerging imaging technique, Cherenkov-excited luminescence scanned tomography (CELST) can recover a high-resolution 3D distribution of quantum emission fields within tissue using X-ray excitation for deep penetrance. However, its reconstruction is an ill-posed and under-conditioned inverse problem because of the diffuse optical emission signal. Deep learning based image reconstruction has shown very good potential for solving these types of problems, however they suffer from a lack of ground-truth image data to confirm when used with experimental data. To overcome this, a self-supervised network cascaded by a 3D reconstruction network and the forward model, termed Selfrec-Net, was proposed to perform CELST reconstruction. Under this framework, the boundary measurements are input to the network to reconstruct the distribution of the quantum field and the predicted measurements are subsequently obtained by feeding the reconstructed result to the forward model. The network was trained by minimizing the loss between the input measurements and the predicted measurements rather than the reconstructed distributions and the corresponding ground truths. Comparative experiments were carried out on both numerical simulations and physical phantoms. For singular luminescent targets, the results demonstrate the effectiveness and robustness of the proposed network, and comparable performance can be attained to a state-of-the-art deep supervised learning algorithm, where the accuracy of the emission yield and localization of the objects was far superior to iterative reconstruction methods. Reconstruction of multiple objects is still reasonable with high localization accuracy, although with limits to the emission yield accuracy as the distribution becomes more complex. Overall though the reconstruction of Selfrec-Net provides a self-supervised way to recover the location and emission yield of molecular distributions in murine model tissues.
Collapse
Affiliation(s)
- Wenqian Zhang
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Ting Hu
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Zhe Li
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Zhonghua Sun
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Kebin Jia
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Huijing Dou
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jinchao Feng
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Brian W. Pogue
- Department of Medical Physics, University of Wisconsin-Madison, Madison, WI 53705, USA
| |
Collapse
|
29
|
Endo K, Yamamoto K, Ohtsuki T. A Denoising Method Using Deep Image Prior to Human-Target Detection Using MIMO FMCW Radar. SENSORS (BASEL, SWITZERLAND) 2022; 22:9401. [PMID: 36502104 PMCID: PMC9739089 DOI: 10.3390/s22239401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Revised: 11/10/2022] [Accepted: 11/28/2022] [Indexed: 06/17/2023]
Abstract
A Multiple-Input Multiple-Output (MIMO) Frequency-Modulated Continuous Wave (FMCW) radar can provide a range-angle map that expresses the signal power against each range and angle. It is possible to estimate object locations by detecting the signal power that exceeds a threshold using an algorithm, such as Constant False Alarm Rate (CFAR). However, noise and multipath components often exist over the range-angle map, which could produce false alarms for an undesired location depending on the threshold setting. In other words, the threshold setting is sensitive in noisy range-angle maps. Therefore, if the noise is reduced, the threshold can be easily set to reduce the number of false alarms. In this paper, we propose a method that improves the CFAR threshold tolerance by denoising a range-angle map using Deep Image Prior (DIP). DIP is an unsupervised deep-learning technique that enables image denoising. In the proposed method, DIP is applied to the range-angle map calculated by the Curve-Length (CL) method, and then the object location is detected over the denoised range-angle map based on Cell-Averaging CFAR (CA-CFAR), which is a typical threshold setting algorithm. Through the experiments to estimate human locations in indoor environments, we confirmed that the proposed method with DIP reduced the number of false alarms and estimated the human location accurately while improving the tolerance of the threshold setting, compared to the method without DIP.
Collapse
Affiliation(s)
- Koji Endo
- Graduate School of Science and Technology, Keio University, Yokohama 223-8522, Japan
| | - Kohei Yamamoto
- Department of Information and Computer Science, Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan
| | - Tomoaki Ohtsuki
- Department of Information and Computer Science, Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan
| |
Collapse
|
30
|
Wijesinghe P, Corsetti S, Chow DJX, Sakata S, Dunning KR, Dholakia K. Experimentally unsupervised deconvolution for light-sheet microscopy with propagation-invariant beams. LIGHT, SCIENCE & APPLICATIONS 2022; 11:319. [PMID: 36319636 PMCID: PMC9626625 DOI: 10.1038/s41377-022-00975-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 08/26/2022] [Accepted: 08/31/2022] [Indexed: 05/25/2023]
Abstract
Deconvolution is a challenging inverse problem, particularly in techniques that employ complex engineered point-spread functions, such as microscopy with propagation-invariant beams. Here, we present a deep-learning method for deconvolution that, in lieu of end-to-end training with ground truths, is trained using known physics of the imaging system. Specifically, we train a generative adversarial network with images generated with the known point-spread function of the system, and combine this with unpaired experimental data that preserve perceptual content. Our method rapidly and robustly deconvolves and super-resolves microscopy images, demonstrating a two-fold improvement in image contrast to conventional deconvolution methods. In contrast to common end-to-end networks that often require 1000-10,000s paired images, our method is experimentally unsupervised and can be trained solely on a few hundred regions of interest. We demonstrate its performance on light-sheet microscopy with propagation-invariant Airy beams in oocytes, preimplantation embryos and excised brain tissue, as well as illustrate its utility for Bessel-beam LSM. This method aims to democratise learned methods for deconvolution, as it does not require data acquisition outwith the conventional imaging protocol.
Collapse
Affiliation(s)
- Philip Wijesinghe
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK.
| | - Stella Corsetti
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK
| | - Darren J X Chow
- Robinson Research Institute, Adelaide Medical School, The University of Adelaide, Adelaide, SA, Australia
- Australian Research Council Centre of Excellence for Nanoscale BioPhotonics, The University of Adelaide, Adelaide, SA, Australia
- Institute for Photonics and Advanced Sensing, The University of Adelaide, Adelaide, SA, Australia
| | - Shuzo Sakata
- Strathclyde Institute of Pharmacy and Biomedical Sciences, University of Strathclyde, Glasgow, G4 0RE, UK
| | - Kylie R Dunning
- Robinson Research Institute, Adelaide Medical School, The University of Adelaide, Adelaide, SA, Australia
- Australian Research Council Centre of Excellence for Nanoscale BioPhotonics, The University of Adelaide, Adelaide, SA, Australia
- Institute for Photonics and Advanced Sensing, The University of Adelaide, Adelaide, SA, Australia
| | - Kishan Dholakia
- SUPA, School of Physics and Astronomy, University of St Andrews, North Haugh, St Andrews, Fife, KY16 9SS, UK.
- School of Biological Sciences, The University of Adelaide, Adelaide, SA, Australia.
- Department of Physics, College of Science, Yonsei University, Seoul, 03722, South Korea.
| |
Collapse
|
31
|
Chen Q, Huang D, Chen R. Fourier ptychographic microscopy with untrained deep neural network priors. OPTICS EXPRESS 2022; 30:39597-39612. [PMID: 36298907 DOI: 10.1364/oe.472171] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Accepted: 09/26/2022] [Indexed: 06/16/2023]
Abstract
We propose a physics-assisted deep neural network scheme in Fourier ptychographic microscopy (FPM) using untrained deep neural network priors (FPMUP) to achieve a high-resolution image reconstruction from multiple low-resolution images. Unlike the traditional training type of deep neural network that requires a large labelled dataset, this proposed scheme does not require training and instead outputs the high-resolution image by optimizing the parameters of neural networks to fit the experimentally measured low-resolution images. Besides the amplitude and phase of the sample function, another two parallel neural networks that generate the general pupil function and illumination intensity factors are incorporated into the carefully designed neural networks, which effectively improves the image quality and robustness when both the aberration and illumination intensity fluctuation are present in FPM. Reconstructions using simulated and experimental datasets are demonstrated, showing that the FPMUP scheme has better image quality than the traditional iterative algorithms, especially for the phase recovery, but at the expense of increasing computational cost. Most importantly, it is found that the FPMUP scheme can predict the Fourier spectrum of the sample outside synthetic aperture of FPM and thus eliminate the ringing effect of the recovered images due to the spectral truncation. Inspired by deep image prior in the field of image processing, we may impute the expansion of Fourier spectrums to the deep prior rooted in the architecture of the careful designed four parallel deep neural networks. We envisage that the resolution of FPM will be further enhanced if the Fourier spectrum of the sample outside the synthetic aperture of FPM is accurately predicted.
Collapse
|
32
|
Baroni A, Bouchama L, Dorizzi B, Gottesman Y. Angularly resolved polarization microscopy for birefringent materials with Fourier ptychography. OPTICS EXPRESS 2022; 30:38984-38994. [PMID: 36258450 DOI: 10.1364/oe.469377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Accepted: 08/11/2022] [Indexed: 06/16/2023]
Abstract
Polarization light microscopy is a very popular approach for structural imaging in optics. So far these methods mainly probe the sample at a fixed angle of illumination. They are consequently only sensitive to the polarization properties along the microscope optical axis. This paper presents a novel method to resolve angularly the polarization properties of birefringent materials, by retrieving quantitatively the spatial variation of their index ellipsoids. Since this method is based on Fourier ptychography microscopy the latter properties are retrieved with a spatial super-resolution factor. An adequate formalism for the Fourier ptychography forward model is introduced to cope with angularly resolved polarization properties. The inverse problem is solved using an unsupervised deep neural network approach that is proven efficient thanks to its performing regularization properties together with its automatic differentiation. Simulated results are reported showing the feasibility of the methods.
Collapse
|
33
|
Ravigopal SR, Brumfiel TA, Sarma A, Desai JP. Fluoroscopic Image-Based 3-D Environment Reconstruction and Automated Path Planning for a Robotically Steerable Guidewire. IEEE Robot Autom Lett 2022; 7:11918-11925. [PMID: 36275193 PMCID: PMC9583954 DOI: 10.1109/lra.2022.3207568] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
Cardiovascular diseases are the leading cause of death globally and surgical treatments for these often begin with the manual placement of a long compliant wire, called a guidewire, through different vasculature. To improve procedure outcomes and reduce radiation exposure, we propose steps towards a fully automated approach for steerable guidewire navigation within vessels. In this paper, we utilize fluoroscopic images to fully reconstruct 3-D printed phantom vasculature models by using a shape-from-silhouette algorithm. The reconstruction is subsequently de-noised using a deep learning-based encoder-decoder network and morphological filtering. This volume is used to model the environment for guidewire traversal. Following this, we present a novel method to plan an optimal path for guidewire traversal in three-dimensional vascular models through the use of slice planes and a modified hybrid A-star algorithm. Finally, the developed reconstruction and planning approaches are applied to an ex vivo porcine aorta, and navigation is demonstrated through the use of a tendon-actuated COaxially Aligned STeerable guidewire (COAST).
Collapse
Affiliation(s)
- Sharan R Ravigopal
- Medical Robotics and Automation (RoboMed) Laboratory, Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA30332 USA
| | - Timothy A Brumfiel
- Medical Robotics and Automation (RoboMed) Laboratory, Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA30332 USA
| | - Achraj Sarma
- Medical Robotics and Automation (RoboMed) Laboratory, Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA30332 USA
| | - Jaydev P Desai
- Medical Robotics and Automation (RoboMed) Laboratory, Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA30332 USA
| |
Collapse
|
34
|
Velasco C, Fletcher TJ, Botnar RM, Prieto C. Artificial intelligence in cardiac magnetic resonance fingerprinting. Front Cardiovasc Med 2022; 9:1009131. [PMID: 36204566 PMCID: PMC9530662 DOI: 10.3389/fcvm.2022.1009131] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Accepted: 08/30/2022] [Indexed: 11/13/2022] Open
Abstract
Magnetic resonance fingerprinting (MRF) is a fast MRI-based technique that allows for multiparametric quantitative characterization of the tissues of interest in a single acquisition. In particular, it has gained attention in the field of cardiac imaging due to its ability to provide simultaneous and co-registered myocardial T1 and T2 mapping in a single breath-held cardiac MRF scan, in addition to other parameters. Initial results in small healthy subject groups and clinical studies have demonstrated the feasibility and potential of MRF imaging. Ongoing research is being conducted to improve the accuracy, efficiency, and robustness of cardiac MRF. However, these improvements usually increase the complexity of image reconstruction and dictionary generation and introduce the need for sequence optimization. Each of these steps increase the computational demand and processing time of MRF. The latest advances in artificial intelligence (AI), including progress in deep learning and the development of neural networks for MRI, now present an opportunity to efficiently address these issues. Artificial intelligence can be used to optimize candidate sequences and reduce the memory demand and computational time required for reconstruction and post-processing. Recently, proposed machine learning-based approaches have been shown to reduce dictionary generation and reconstruction times by several orders of magnitude. Such applications of AI should help to remove these bottlenecks and speed up cardiac MRF, improving its practical utility and allowing for its potential inclusion in clinical routine. This review aims to summarize the latest developments in artificial intelligence applied to cardiac MRF. Particularly, we focus on the application of machine learning at different steps of the MRF process, such as sequence optimization, dictionary generation and image reconstruction.
Collapse
Affiliation(s)
- Carlos Velasco
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- *Correspondence: Carlos Velasco
| | - Thomas J. Fletcher
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - René M. Botnar
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- Institute for Biological and Medical Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile
- Millennium Institute for Intelligent Healthcare Engineering, Santiago, Chile
| | - Claudia Prieto
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- Institute for Biological and Medical Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile
- Millennium Institute for Intelligent Healthcare Engineering, Santiago, Chile
| |
Collapse
|
35
|
Monroy B, Bacca J, Arguello H. JR2net: a joint non-linear representation and recovery network for compressive spectral imaging. APPLIED OPTICS 2022; 61:7757-7766. [PMID: 36256378 DOI: 10.1364/ao.463726] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 08/17/2022] [Indexed: 06/16/2023]
Abstract
Deep learning models are state-of-the-art in compressive spectral imaging (CSI) recovery. These methods use a deep neural network (DNN) as an image generator to learn non-linear mapping from compressed measurements to the spectral image. For instance, the deep spectral prior approach uses a convolutional autoencoder (CAE) network in the optimization algorithm to recover the spectral image by using a non-linear representation. However, the CAE training is detached from the recovery problem, which does not guarantee optimal representation of the spectral images for the CSI problem. This work proposes a joint non-linear representation and recovery network (JR2net), linking the representation and recovery task into a single optimization problem. JR2net consists of an optimization-inspired network following an alternating direction method of multipliers (ADMM) formulation that learns a non-linear low-dimensional representation and simultaneously performs the spectral image recovery, trained via the end-to-end approach. Experimental results show the superiority of the proposed method with improvements up to 2.57 dB in peak signal-to-noise ratio (PSNR) and performance around 2000 times faster than state-of-the-art methods.
Collapse
|
36
|
Gan Y, Gao F, Dong J, Chen S. Arbitrary-Scale Texture Generation From Coarse-Grained Control. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2022; 31:5841-5855. [PMID: 36054394 DOI: 10.1109/tip.2022.3201710] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Existing deep-network based texture synthesis approaches all focus on fine-grained control of texture generation by synthesizing images from exemplars. Since the networks employed by most of these methods are always tied to individual exemplar textures, a large number of individual networks have to be trained when modeling various textures. In this paper, we propose to generate textures directly from coarse-grained control or high-level guidance, such as texture categories, perceptual attributes and semantic descriptions. We fulfill the task by parsing the generation process of a texture into the three-level Bayesian hierarchical model. A coarse-grained signal first determines a distribution over Markov random fields. Then a Markov random field is used to model the distribution of the final output textures. Finally, an output texture is generated from the sampled Markov random field distribution. At the bottom level of the Bayesian hierarchy, the isotropic and ergodic characteristics of the textures favor a construction that consists of a fully convolutional network. The proposed method integrates texture creation and texture synthesis into one pipeline for real-time texture generation, and enables users to readily obtain diverse textures with arbitrary scales from high-level guidance only. Extensive experiments demonstrate that the proposed method is capable of generating plausible textures that are faithful to user-defined control, and achieving impressive texture metamorphosis by interpolation in the learned texture manifold.
Collapse
|
37
|
Ke R, Schonlieb CB. Unsupervised Image Restoration Using Partially Linear Denoisers. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:5796-5812. [PMID: 33819148 DOI: 10.1109/tpami.2021.3070382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term and a small residual term. We show that these denoisers can be trained with only noisy images under the condition that the noise has zero mean and known variance. The exact distribution of the noise, however, is not assumed to be known. We show the superiority of our approach for image denoising, and demonstrate its extension to solving other restoration problems such as image deblurring where the ground truth is not available. Our method outperforms some recent unsupervised and self-supervised deep denoising models that do not require clean images for their training. For deblurring problems, the method, using only one noisy and blurry observation per image, reaches a quality not far away from its fully supervised counterparts on a benchmark dataset.
Collapse
|
38
|
Fourcade C, Ferrer L, Moreau N, Santini G, Brennan A, Rousseau C, Lacombe M, Fleury V, Colombié M, Jézéquel P, Rubeaux M, Mateus D. Deformable image registration with deep network priors: a study on longitudinal PET images. Phys Med Biol 2022; 67. [DOI: 10.1088/1361-6560/ac7e17] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Accepted: 07/04/2022] [Indexed: 11/12/2022]
Abstract
Abstract
Objective. This paper proposes a novel approach for the longitudinal registration of PET imaging acquired for the monitoring of patients with metastatic breast cancer. Unlike with other image analysis tasks, the use of deep learning (DL) has not significantly improved the performance of image registration. With this work, we propose a new registration approach to bridge the performance gap between conventional and DL-based methods: medical image registration method regularized by architecture (MIRRBA). Approach.
MIRRBA is a subject-specific deformable registration method which relies on a deep pyramidal architecture to parametrize the deformation field. Diverging from the usual deep-learning paradigms, MIRRBA does not require a learning database, but only a pair of images to be registered that is used to optimize the network's parameters. We applied MIRRBA on a private dataset of 110 whole-body PET images of patients with metastatic breast cancer. We used different architecture configurations to produce the deformation field and studied the results obtained. We also compared our method to several standard registration approaches: two conventional iterative registration methods (ANTs and Elastix) and two supervised DL-based models (LapIRN and Voxelmorph). Registration accuracy was evaluated using the Dice score, the target registration error, the average Hausdorff distance and the detection rate, while the realism of the registration obtained was evaluated using Jacobian's determinant. The ability of the different methods to shrink disappearing lesions was also computed with the disappearing rate. Main results. MIRRBA significantly improved all metrics when compared to DL-based approaches. The organ and lesion Dice scores of Voxelmorph improved by 6% and 52% respectively, while the ones of LapIRN increased by 5% and 65%. Regarding conventional approaches, MIRRBA presented comparable results showing the feasibility of our method. Significance. In this paper, we also demonstrate the regularizing power of deep architectures and present new elements to understand the role of the architecture in DL methods used for registration.
Collapse
|
39
|
Artificial intelligence-based PET image acquisition and reconstruction. Clin Transl Imaging 2022. [DOI: 10.1007/s40336-022-00508-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
40
|
Guo Y, Jiao M, Wan G, Xiang J, Wang S, Liu F. EEG Source Imaging using GANs with Deep Image Prior. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:572-575. [PMID: 36083924 DOI: 10.1109/embc48229.2022.9871172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Brain source localization from electroencephalogram (EEG) signals is an challenging problem for noninvasively localizing the brain activity. Conventional methods use handcrafted regularization terms based on neural-physiological assumptions by exploiting the spatial-temporal structure on the source signals. In recent years, deep learning frameworks have demonstrated superior performance for solving the inverse problems in the natural and medical imaging field. This study proposes a novel unsupervised learning training-free framework based on Generative Adversarial Networks and deep image prior (GANs-DIP) as a generative model simulating spatially structured source signal. The proposed framework can faithfully recover extended source patches activation patterns of the brain in an unsupervised manner. Numerical experiments on a realistic brain model are performed under different levels of signal-to-noise ratio (SNR). The proposed model shows satisfactory performance in recovering the underlying source activation.
Collapse
|
41
|
Nishii T, Kobayashi T, Tanaka H, Kotoku A, Ohta Y, Morita Y, Umehara K, Ota J, Horinouchi H, Ishida T, Fukuda T. Deep Learning-based Post Hoc CT Denoising for Myocardial Delayed Enhancement. Radiology 2022; 305:82-91. [PMID: 35762889 DOI: 10.1148/radiol.220189] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Background To improve myocardial delayed enhancement (MDE) CT, a deep learning (DL)-based post hoc denoising method supervised with averaged MDE CT data was developed. Purpose To assess the image quality of denoised MDE CT images and evaluate their diagnostic performance by using late gadolinium enhancement (LGE) MRI as a reference. Materials and methods MDE CT data obtained by averaging three acquisitions with a single breath hold 5 minutes after the contrast material injection in patients from July 2020 to October 2021 were retrospectively reviewed. Preaveraged images obtained in 100 patients as inputs and averaged images as ground truths were used to supervise a residual dense network (RDN). The original single-shot image, standard averaged image, RDN-denoised original (DLoriginal) image, and RDN-denoised averaged (DLave) image of holdout cases were compared. In 40 patients, the CT value and image noise in the left ventricular cavity and myocardium were assessed. The segmental presence of MDE in the remaining 40 patients who underwent reference LGE MRI was evaluated. The sensitivity, specificity, and accuracy of each type of CT image and the improvement in accuracy achieved with the RDN were assessed using odds ratios (ORs) estimated with the generalized estimation equation. Results Overall, 180 patients (median age, 66 years [IQR, 53-74 years]; 107 men) were included. The RDN reduced image noise to 28% of the original level while maintaining equivalence in the CT values (P < .001 for all). The sensitivity, specificity, and accuracy of the original images were 77.9%, 84.4%, and 82.3%, of the averaged images were 89.7%, 87.9%, and 88.5%, of the DLoriginal images were 93.1%, 87.5%, and 89.3%, and of the DLave images were 95.1%, 93.1%, and 93.8%, respectively. DLoriginal images showed improved accuracy compared with the original images (OR, 1.8 [95% CI: 1.2, 2.9]; P = .011) and DLave images showed improved accuracy compared with the averaged images (OR, 2.0 [95% CI: 1.2, 3.5]; P = .009). Conclusion The proposed denoising network supervised with averaged CT images reduced image noise and improved the diagnostic performance for myocardial delayed enhancement CT. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Vannier and Wang in this issue.
Collapse
Affiliation(s)
- Tatsuya Nishii
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Takuma Kobayashi
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Hironori Tanaka
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Akiyuki Kotoku
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yasutoshi Ohta
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yoshiaki Morita
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Kensuke Umehara
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Junko Ota
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Hiroki Horinouchi
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Takayuki Ishida
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Tetsuya Fukuda
- From the Department of Radiology, National Cerebral and Cardiovascular Center, 6-1 Kishibe-shinmachi, Suita 564-8565, Japan (T.N., T.K., H.T., A.K., Y.O., Y.M., H.H., T.F.); Department of Medical Physics and Engineering, Graduate School of Medicine, Osaka University, Suita, Japan (T.K., K.U., J.O., T.I.); Medical Informatics Section, QST Hospital (K.U., J.O.), and Applied MRI Research, Department of Molecular Imaging and Theranostics, Institute for Quantum Medical Science (K.U., J.O.), National Institutes for Quantum Science and Technology, Chiba, Japan
| |
Collapse
|
42
|
Wu C, Lorenzo G, Hormuth DA, Lima EABF, Slavkova KP, DiCarlo JC, Virostko J, Phillips CM, Patt D, Chung C, Yankeelov TE. Integrating mechanism-based modeling with biomedical imaging to build practical digital twins for clinical oncology. BIOPHYSICS REVIEWS 2022; 3:021304. [PMID: 35602761 PMCID: PMC9119003 DOI: 10.1063/5.0086789] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 04/29/2022] [Indexed: 12/11/2022]
Abstract
Digital twins employ mathematical and computational models to virtually represent a physical object (e.g., planes and human organs), predict the behavior of the object, and enable decision-making to optimize the future behavior of the object. While digital twins have been widely used in engineering for decades, their applications to oncology are only just emerging. Due to advances in experimental techniques quantitatively characterizing cancer, as well as advances in the mathematical and computational sciences, the notion of building and applying digital twins to understand tumor dynamics and personalize the care of cancer patients has been increasingly appreciated. In this review, we present the opportunities and challenges of applying digital twins in clinical oncology, with a particular focus on integrating medical imaging with mechanism-based, tissue-scale mathematical modeling. Specifically, we first introduce the general digital twin framework and then illustrate existing applications of image-guided digital twins in healthcare. Next, we detail both the imaging and modeling techniques that provide practical opportunities to build patient-specific digital twins for oncology. We then describe the current challenges and limitations in developing image-guided, mechanism-based digital twins for oncology along with potential solutions. We conclude by outlining five fundamental questions that can serve as a roadmap when designing and building a practical digital twin for oncology and attempt to provide answers for a specific application to brain cancer. We hope that this contribution provides motivation for the imaging science, oncology, and computational communities to develop practical digital twin technologies to improve the care of patients battling cancer.
Collapse
Affiliation(s)
- Chengyue Wu
- Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas 78712, USA
| | | | | | | | - Kalina P. Slavkova
- Department of Physics, The University of Texas at Austin, Austin, Texas 78712, USA
| | | | | | - Caleb M. Phillips
- Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas 78712, USA
| | - Debra Patt
- Texas Oncology, Austin, Texas 78731, USA
| | - Caroline Chung
- Department of Radiation Oncology, MD Anderson Cancer Center, University of Texas, Houston, Texas 77030, USA
| | | |
Collapse
|
43
|
Deep Learning-Based Denoising in Brain Tumor CHO PET: Comparison with Traditional Approaches. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12105187] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
Abstract
18F-choline (CHO) PET image remains noisy despite minimum physiological activity in the normal brain, and this study developed a deep learning-based denoising algorithm for brain tumor CHO PET. Thirty-nine presurgical CHO PET/CT data were retrospectively collected for patients with pathological confirmed primary diffuse glioma. Two conventional denoising methods, namely, block-matching and 3D filtering (BM3D) and non-local means (NLM), and two deep learning-based approaches, namely, Noise2Noise (N2N) and Noise2Void (N2V), were established for imaging denoising, and the methods were developed without paired data. All algorithms improved the image quality to a certain extent, with the N2N demonstrating the best contrast-to-noise ratio (CNR) (4.05 ± 3.45), CNR improvement ratio (13.60% ± 2.05%) and the lowest entropy (1.68 ± 0.17), compared with other approaches. Little changes were identified in traditional tumor PET features including maximum standard uptake value (SUVmax), SUVmean and total lesion activity (TLA), while the tumor-to-normal (T/N ratio) increased thanks to smaller noise. These results suggested that the N2N algorithm can acquire sufficient denoising performance while preserving the original features of tumors, and may be generalized for abundant brain tumor PET images.
Collapse
|
44
|
Chen M, Quan Y, Pang T, Ji H. Nonblind Image Deconvolution via Leveraging Model Uncertainty in An Untrained Deep Neural Network. Int J Comput Vis 2022. [DOI: 10.1007/s11263-022-01621-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
|
45
|
Jung W, Kim EH, Ko J, Jeong G, Choi MH. Convolutional neural network-based reconstruction for acceleration of prostate T 2 weighted MR imaging: a retro- and prospective study. Br J Radiol 2022; 95:20211378. [PMID: 35148172 PMCID: PMC10993971 DOI: 10.1259/bjr.20211378] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Revised: 01/19/2022] [Accepted: 01/24/2022] [Indexed: 11/05/2022] Open
Abstract
OBJECTIVE The aim of this study was to develop a deep neural network (DNN)-based parallel imaging reconstruction for highly accelerated 2D turbo spin echo (TSE) data in prostate MRI without quality degradation compared to conventional scans. METHODS 155 participant data were acquired for training and testing. Two DNN models were generated according to the number of acquisitions (NAQ) of input images: DNN-N1 for NAQ = 1 and DNN-N2 for NAQ = 2. In the test data, DNN and TSE images were compared by quantitative error metrics. The visual appropriateness of DNN reconstructions on accelerated scans (DNN-N1 and DNN-N2) and conventional scans (TSE-Conv) was assessed for nine parameters by two radiologists. The lesion detection was evaluated at DNNs and TES-Conv by prostate imaging-reporting and data system. RESULTS The scan time was reduced by 71% at NAQ = 1, and 42% at NAQ = 2. Quantitative evaluation demonstrated the better error metrics of DNN images (29-43% lower NRMSE, 4-13% higher structure similarity index, and 2.8-4.8 dB higher peak signal-to-noise ratio; p < 0.001) than TSE images. In the assessment of the visual appropriateness, both radiologists evaluated that DNN-N2 showed better or comparable performance in all parameters compared to TSE-Conv. In the lesion detection, DNN images showed almost perfect agreement (κ > 0.9) scores with TSE-Conv. CONCLUSIONS DNN-based reconstruction in highly accelerated prostate TSE imaging showed comparable quality to conventional TSE. ADVANCES IN KNOWLEDGE Our framework reduces the scan time by 42% of conventional prostate TSE imaging without sequence modification, revealing great potential for clinical application.
Collapse
Affiliation(s)
| | - Eu Hyun Kim
- Department of Radiology, St.Vincent’s Hospital, College
of Medicine, The Catholic University of Korea, Suwon,
Gyeonggi-do, Republic of Korea
| | - Jingyu Ko
- AIRS Medical, Seoul,
Republic of Korea
| | | | - Moon Hyung Choi
- Department of Radiology, Eunpyeong St. Mary’s Hospital,
College of Medicine, The Catholic University of Korea,
Seoul, Republic of Korea
| |
Collapse
|
46
|
Jing J, Xia W, Hou M, Chen H, Liu Y, Zhou J, Zhang Y. Training low dose CT denoising network without high quality reference data. Phys Med Biol 2022; 67. [DOI: 10.1088/1361-6560/ac5f70] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2021] [Accepted: 03/21/2022] [Indexed: 11/12/2022]
Abstract
Abstract
Objective. Currently, the field of low-dose CT (LDCT) denoising is dominated by supervised learning based methods, which need perfectly registered pairs of LDCT and its corresponding clean reference image (normal-dose CT). However, training without clean labels is more practically feasible and significant, since it is clinically impossible to acquire a large amount of these paired samples. In this paper, a self-supervised denoising method is proposed for LDCT imaging. Approach. The proposed method does not require any clean images. In addition, the perceptual loss is used to achieve data consistency in feature domain during the denoising process. Attention blocks used in decoding phase can help further improve the image quality. Main results. In the experiments, we validate the effectiveness of our proposed self-supervised framework and compare our method with several state-of-the-art supervised and unsupervised methods. The results show that our proposed model achieves competitive performance in both qualitative and quantitative aspects to other methods. Significance. Our framework can be directly applied to most denoising scenarios without collecting pairs of training data, which is more flexible for real clinical scenario.
Collapse
|
47
|
Deep Internal Learning for Inpainting of Cloud-Affected Regions in Satellite Imagery. REMOTE SENSING 2022. [DOI: 10.3390/rs14061342] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
Cloud cover remains a significant limitation to a broad range of applications relying on optical remote sensing imagery, including crop identification/yield prediction, climate monitoring, and land cover classification. A common approach to cloud removal treats the problem as an inpainting task and imputes optical data in the cloud-affected regions employing either mosaicing historical data or making use of sensing modalities not impacted by cloud obstructions, such as SAR. Recently, deep learning approaches have been explored in these applications; however, the majority of reported solutions rely on external learning practices, i.e., models trained on fixed datasets. Although these models perform well within the context of a particular dataset, a significant risk of spatial and temporal overfitting exists when applied in different locations or at different times. Here, cloud removal was implemented within an internal learning regime through an inpainting technique based on the deep image prior. The approach was evaluated on both a synthetic dataset with an exact ground truth, as well as real samples. The ability to inpaint the cloud-affected regions for varying weather conditions across a whole year with no prior training was demonstrated, and the performance of the approach was characterised.
Collapse
|
48
|
Enßlin T. Information Field Theory and Artificial Intelligence. ENTROPY 2022; 24:e24030374. [PMID: 35327885 PMCID: PMC8947090 DOI: 10.3390/e24030374] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/18/2021] [Revised: 02/28/2022] [Accepted: 03/04/2022] [Indexed: 12/10/2022]
Abstract
Information field theory (IFT), the information theory for fields, is a mathematical framework for signal reconstruction and non-parametric inverse problems. Artificial intelligence (AI) and machine learning (ML) aim at generating intelligent systems, including such for perception, cognition, and learning. This overlaps with IFT, which is designed to address perception, reasoning, and inference tasks. Here, the relation between concepts and tools in IFT and those in AI and ML research are discussed. In the context of IFT, fields denote physical quantities that change continuously as a function of space (and time) and information theory refers to Bayesian probabilistic logic equipped with the associated entropic information measures. Reconstructing a signal with IFT is a computational problem similar to training a generative neural network (GNN) in ML. In this paper, the process of inference in IFT is reformulated in terms of GNN training. In contrast to classical neural networks, IFT based GNNs can operate without pre-training thanks to incorporating expert knowledge into their architecture. Furthermore, the cross-fertilization of variational inference methods used in IFT and ML are discussed. These discussions suggest that IFT is well suited to address many problems in AI and ML research and application.
Collapse
Affiliation(s)
- Torsten Enßlin
- Max Planck Institute for Astrophysics, Karl-Schwarzschild-Strasse 1, 85748 Garching, Germany;
- Physics Department, Ludwig-Maximilians-Universität München, Geschwister-Scholl-Platz 1, 80539 Munich, Germany
| |
Collapse
|
49
|
Yokota T, Hontani H, Zhao Q, Cichocki A. Manifold Modeling in Embedded Space: An Interpretable Alternative to Deep Image Prior. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; 33:1022-1036. [PMID: 33275587 DOI: 10.1109/tnnls.2020.3037923] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Deep image prior (DIP), which uses a deep convolutional network (ConvNet) structure as an image prior, has attracted wide attention in computer vision and machine learning. DIP empirically shows the effectiveness of the ConvNet structures for various image restoration applications. However, why the DIP works so well is still unknown. In addition, the reason why the convolution operation is useful in image reconstruction, or image enhancement is not very clear. This study tackles this ambiguity of ConvNet/DIP by proposing an interpretable approach that divides the convolution into "delay embedding" and "transformation" (i.e., encoder-decoder). Our approach is a simple, but essential, image/tensor modeling method that is closely related to self-similarity. The proposed method is called manifold modeling in embedded space (MMES) since it is implemented using a denoising autoencoder in combination with a multiway delay-embedding transform. In spite of its simplicity, MMES can obtain quite similar results to DIP on image/tensor completion, super-resolution, deconvolution, and denoising. In addition, MMES is proven to be competitive with DIP, as shown in our experiments. These results can also facilitate interpretation/characterization of DIP from the perspective of a "low-dimensional patch-manifold prior."
Collapse
|
50
|
Noise-Resistant Demosaicing with Deep Image Prior Network and Random RGBW Color Filter Array. SENSORS 2022; 22:s22051767. [PMID: 35270912 PMCID: PMC8914803 DOI: 10.3390/s22051767] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/20/2022] [Revised: 02/07/2022] [Accepted: 02/21/2022] [Indexed: 02/01/2023]
Abstract
In this paper, we propose a deep-image-prior-based demosaicing method for a random RGBW color filter array (CFA). The color reconstruction from the random RGBW CFA is performed by the deep image prior network, which uses only the RGBW CFA image as the training data. To our knowledge, this work is a first attempt to reconstruct the color image with a neural network using only a single RGBW CFA in the training. Due to the White pixels in the RGBW CFA, more light is transmitted through the CFA than in the case with the conventional RGB CFA. As the image sensor can detect more light, the signal-to-noise-ratio (SNR) increases and the proposed demosaicing method can reconstruct the color image with a higher visual quality than other existing demosaicking methods, especially in the presence of noise. We propose a loss function that can train the deep image prior (DIP) network to reconstruct the colors from the White pixels as well as from the red, green, and blue pixels in the RGBW CFA. Apart from using the DIP network, no additional complex reconstruction algorithms are required for the demosaicing. The proposed demosaicing method becomes useful in situations when the noise becomes a major problem, for example, in low light conditions. Experimental results show the validity of the proposed method for joint demosaicing and denoising.
Collapse
|