Structurally-sensitive Multi-scale Deep Neural Network for Low-Dose CT Denoising

A Deep Learning Framework for Segmenting Brain Tumors Using MRI and Synthetically Generated CT Images

Multi-modal three-dimensional (3-D) image segmentation is used in many medical applications, such as disease diagnosis, treatment planning, and image-guided surgery. Although multi-modal images provide information that no single image modality alone can provide, integrating such information to be used in segmentation is a challenging task. Numerous methods have been introduced to solve the problem of multi-modal medical image segmentation in recent years. In this paper, we propose a solution for the task of brain tumor segmentation. To this end, we first introduce a method of enhancing an existing magnetic resonance imaging (MRI) dataset by generating synthetic computed tomography (CT) images. Then, we discuss a process of systematic optimization of a convolutional neural network (CNN) architecture that uses this enhanced dataset, in order to customize it for our task. Using publicly available datasets, we show that the proposed method outperforms similar existing methods.

Projection decomposition via univariate optimization for dual-energy CT

Dual-energy computed tomography (DECT) acquires two x-ray projection datasets with different x-ray energy spectra, performs material-specific image reconstruction based on the energy-dependent non-linear integral model, and provides more accurate quantification of attenuation coefficients than single energy spectrum CT. In the diagnostic energy range, x-ray energy-dependent attenuation is mainly caused by photoelectric absorption and Compton scattering. Theoretically, these two physical components of the x-ray attenuation mechanism can be determined from two projection datasets with distinct energy spectra. Practically, the solution of the non-linear integral equation is complicated due to spectral uncertainty, detector sensitivity, and data noise. Conventional multivariable optimization methods are prone to local minima. In this paper, we develop a new method for DECT image reconstruction in the projection domain. This method combines an analytic solution of a polynomial equation and a univariate optimization to solve the polychromatic non-linear integral equation. The polynomial equation of an odd order has a unique real solution with sufficient accuracy for image reconstruction, and the univariate optimization can achieve the global optimal solution, allowing accurate and stable projection decomposition for DECT. Numerical and physical phantom experiments are performed to demonstrate the effectiveness of the method in comparison with the state-of-the-art projection decomposition methods. As a result, the univariate optimization method yields a quality improvement of 15% for image reconstruction and substantial reduction of the computational time, as compared to the multivariable optimization methods.

Artifact-Assisted multi-level and multi-scale feature fusion attention network for low-dose CT denoising

BACKGROUND AND OBJECTIVE

Since low-dose computed tomography (LDCT) images typically have higher noise that may affect accuracy of disease diagnosis, the objective of this study is to develop and evaluate a new artifact-assisted feature fusion attention (AAFFA) network to extract and reduce image artifact and noise in LDCT images.

METHODS

In AAFFA network, a feature fusion attention block is constructed for local multi-scale artifact feature extraction and progressive fusion from coarse to fine. A multi-level fusion architecture based on skip connection and attention modules is also introduced for artifact feature extraction. Specifically, long-range skip connections are used to enhance and fuse artifact features with different depth levels. Then, the fused shallower features enter channel attention for better extraction of artifact features, and the fused deeper features are sent into pixel attention for focusing on the artifact pixel information. Besides, an artifact channel is designed to provide rich artifact features and guide the extraction of noise and artifact features. The AAPM LDCT Challenge dataset is used to train and test the network. The performance is evaluated by using both visual observation and quantitative metrics including peak signal-noise-ratio (PSNR), structural similarity index (SSIM) and visual information fidelity (VIF).

RESULTS

Using AAFFA network improves the averaged PSNR/SSIM/VIF values of AAPM LDCT images from 43.4961, 0.9595, 0.3926 to 48.2513, 0.9859, 0.4589, respectively.

CONCLUSIONS

The proposed AAFFA network is able to effectively reduce noise and artifacts while preserving object edges. Assessment of visual quality and quantitative index demonstrates the significant improvement compared with other image denoising methods.

Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis

Fusing multi-modality medical images, such as magnetic resonance (MR) imaging and positron emission tomography (PET), can provide various anatomical and functional information about the human body. However, PET data is not always available for several reasons, such as high cost, radiation hazard, and other limitations. This paper proposes a 3D end-to-end synthesis network called Bidirectional Mapping Generative Adversarial Networks (BMGAN). Image contexts and latent vectors are effectively used for brain MR-to-PET synthesis. Specifically, a bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high-dimensional latent space. Moreover, the 3D Dense-UNet generator architecture and the hybrid loss functions are further constructed to improve the visual quality of cross-modality synthetic images. The most appealing part is that the proposed method can synthesize perceptually realistic PET images while preserving the diverse brain structures of different subjects. Experimental results demonstrate that the performance of the proposed method outperforms other competitive methods in terms of quantitative measures, qualitative displays, and evaluation metrics for classification.

Noise Conscious Training of Non Local Neural Network Powered by Self Attentive Spectral Normalized Markovian Patch GAN for Low Dose CT Denoising

The explosive rise of the use of Computer tomography (CT) imaging in medical practice has heightened public concern over the patient's associated radiation dose. On the other hand, reducing the radiation dose leads to increased noise and artifacts, which adversely degrades the scan's interpretability. In recent times, the deep learning-based technique has emerged as a promising method for low dose CT(LDCT) denoising. However, some common bottleneck still exists, which hinders deep learning-based techniques from furnishing the best performance. In this study, we attempted to mitigate these problems with three novel accretions. First, we propose a novel convolutional module as the first attempt to utilize neighborhood similarity of CT images for denoising tasks. Our proposed module assisted in boosting the denoising by a significant margin. Next, we moved towards the problem of non-stationarity of CT noise and introduced a new noise aware mean square error loss for LDCT denoising. The loss mentioned above also assisted to alleviate the laborious effort required while training CT denoising network using image patches. Lastly, we propose a novel discriminator function for CT denoising tasks. The conventional vanilla discriminator tends to overlook the fine structural details and focus on the global agreement. Our proposed discriminator leverage self-attention and pixel-wise GANs for restoring the diagnostic quality of LDCT images. Our method validated on a publicly available dataset of the 2016 NIH-AAPM-Mayo Clinic Low Dose CT Grand Challenge performed remarkably better than the existing state of the art method. The corresponding source code is available at: https://github.com/reach2sbera/ldct_nonlocal.

Automatic Inter-Frame Patient Motion Correction for Dynamic Cardiac PET Using Deep Learning

Patient motion during dynamic PET imaging can induce errors in myocardial blood flow (MBF) estimation. Motion correction for dynamic cardiac PET is challenging because the rapid tracer kinetics of 82Rb leads to substantial tracer distribution change across different dynamic frames over time, which can cause difficulties for image registration-based motion correction, particularly for early dynamic frames. In this paper, we developed an automatic deep learning-based motion correction (DeepMC) method for dynamic cardiac PET. In this study we focused on the detection and correction of inter-frame rigid translational motion caused by voluntary body movement and pattern change of respiratory motion. A bidirectional-3D LSTM network was developed to fully utilize both local and nonlocal temporal information in the 4D dynamic image data for motion detection. The network was trained and evaluated over motion-free patient scans with simulated motion so that the motion ground-truths are available, where one million samples based on 65 patient scans were used in training, and 600 samples based on 20 patient scans were used in evaluation. The proposed method was also evaluated using additional 10 patient datasets with real motion. We demonstrated that the proposed DeepMC obtained superior performance compared to conventional registration-based methods and other convolutional neural networks (CNN), in terms of motion estimation and MBF quantification accuracy. Once trained, DeepMC is much faster than the registration-based methods and can be easily integrated into the clinical workflow. In the future work, additional investigation is needed to evaluate this approach in a clinical context with realistic patient motion.

High-Resolution Magnetic Resonance Spectroscopic Imaging using a Multi-Encoder Attention U-Net with Structural and Adversarial Loss

Common to most medical imaging techniques, the spatial resolution of Magnetic Resonance Spectroscopic Imaging (MRSI) is ultimately limited by the achievable SNR. This work presents a deep learning method for 1H-MRSI spatial resolution enhancement, based on the observation that multi-parametric MRI images provide relevant spatial priors for MRSI enhancement. A Multi-encoder Attention U-Net (MAU-Net) architecture was constructed to process a MRSI metabolic map and three different MRI modalities through separate encoding paths. Spatial attention modules were incorporated to automatically learn spatial weights that highlight salient features for each MRI modality. MAU-Net was trained based on in vivo brain imaging data from patients with high-grade gliomas, using a combined loss function consisting of pixel, structural and adversarial loss. Experimental results showed that the proposed method is able to reconstruct high-quality metabolic maps with a high-resolution of 64×64 from a low-resolution of 16 × 16, with better performance compared to several baseline methods.

Deep Interactive Denoiser (DID) for X-Ray Computed Tomography

Low-dose computed tomography (LDCT) is desirable for both diagnostic imaging and image-guided interventions. Denoisers are widely used to improve the quality of LDCT. Deep learning (DL)-based denoisers have shown state-of-the-art performance and are becoming mainstream methods. However, there are two challenges to using DL-based denoisers: 1) a trained model typically does not generate different image candidates with different noise-resolution tradeoffs, which are sometimes needed for different clinical tasks; and 2) the model's generalizability might be an issue when the noise level in the testing images differs from that in the training dataset. To address these two challenges, in this work, we introduce a lightweight optimization process that can run on top of any existing DL-based denoiser during the testing phase to generate multiple image candidates with different noise-resolution tradeoffs suitable for different clinical tasks in real time. Consequently, our method allows users to interact with the denoiser to efficiently review various image candidates and quickly pick the desired one; thus, we termed this method deep interactive denoiser (DID). Experimental results demonstrated that DID can deliver multiple image candidates with different noise-resolution tradeoffs and shows great generalizability across various network architectures, as well as training and testing datasets with various noise levels.

Variance-aware attention U-Net for multi-organ segmentation

PURPOSE

With the continuous development of deep learning based medical image segmentation technology, it is expected to attain more robust and accurate performance for more challenging tasks, such as multi-organs, small/irregular areas, and ambiguous boundary issues.

METHODS

We propose a variance-aware attention U-Net to solve the problem of multi-organ segmentation. Specifically, a simple yet effective variance-based uncertainty mechanism is devised to evaluate the discrimination of each voxel via its prediction probability. The proposed variance uncertainty is further embedded into an attention architecture, which not only aggregates multi-level deep features in a global-level but also enforces the network to pay extra attention to voxels with uncertain predictions during training.

RESULTS

Extensive experiments on challenging abdominal multi-organ CT dataset show that our proposed method consistently outperforms cutting-edge attention networks with respect to the evaluation metrics of Dice index (DSC), 95% Hausdorff distance (95HD) and average symmetric surface distance (ASSD).

CONCLUSIONS

The proposed network provides an accurate and robust solution for multi-organ segmentation and has the potential to be used for improving other segmentation applications.

The feasibility of deep learning-based synthetic contrast-enhanced CT from nonenhanced CT in emergency department patients with acute abdominal pain

Our objective was to investigate the feasibility of deep learning-based synthetic contrast-enhanced CT (DL-SCE-CT) from nonenhanced CT (NECT) in patients who visited the emergency department (ED) with acute abdominal pain (AAP). We trained an algorithm generating DL-SCE-CT using NECT with paired precontrast/postcontrast images. For clinical application, 353 patients from three institutions who visited the ED with AAP were included. Six reviewers (experienced radiologists, ER1-3; training radiologists, TR1-3) made diagnostic and disposition decisions using NECT alone and then with NECT and DL-SCE-CT together. The radiologists’ confidence in decisions was graded using a 5-point scale. The diagnostic accuracy using DL-SCE-CT improved in three radiologists (50%, P = 0.023, 0.012, < 0.001, especially in 2/3 of TRs). The confidence of diagnosis and disposition improved significantly in five radiologists (83.3%, P < 0.001). Particularly, in subgroups with underlying malignancy and miscellaneous medical conditions (MMCs) and in CT-negative cases, more radiologists reported increased confidence in diagnosis (83.3% [5/6], 100.0% [6/6], and 83.3% [5/6], respectively) and disposition (66.7% [4/6], 83.3% [5/6] and 100% [6/6], respectively). In conclusion, DL-SCE-CT enhances the accuracy and confidence of diagnosis and disposition regarding patients with AAP in the ED, especially for less experienced radiologists, in CT-negative cases, and in certain disease subgroups with underlying malignancy and MMCs.

Comparison of convolutional neural networks for detecting large vessel occlusion on computed tomography angiography

PURPOSE

Artificial intelligence diagnosis and triage of large vessel occlusion may quicken clinical response for a subset of time-sensitive acute ischemic stroke patients, improving outcomes. Differences in architectural elements within data-driven convolutional neural network (CNN) models impact performance. Foreknowledge of effective model architectural elements for domain-specific problems can narrow the search for candidate models and inform strategic model design and adaptation to optimize performance on available data. Here, we study CNN architectures with a range of learnable parameters and which span the inclusion of architectural elements, such as parallel processing branches and residual connections with varying methods of recombining residual information.

METHODS

We compare five CNNs: ResNet-50, DenseNet-121, EfficientNet-B0, PhiNet, and an Inception module-based network, on a computed tomography angiography large vessel occlusion detection task. The models were trained and preliminarily evaluated with 10-fold cross-validation on preprocessed scans (n = 240). An ablation study was performed on PhiNet due to superior cross-validated test performance across accuracy, precision, recall, specificity, and F1 score. The final evaluation of all models was performed on a withheld external validation set (n = 60) and these predictions were subsequently calibrated with sigmoid curves.

RESULTS

Uncalibrated results on the withheld external validation set show that DenseNet-121 had the best average performance on accuracy, precision, recall, specificity, and F1 score. After calibration DenseNet-121 maintained superior performance on all metrics except recall.

CONCLUSIONS

The number of learnable parameters in our five models and best-ablated PhiNet directly related to cross-validated test performance-the smaller the model the better. However, this pattern did not hold when looking at generalization on the withheld external validation set. DenseNet-121 generalized the best; we posit this was due to its heavy use of residual connections utilizing concatenation, which causes feature maps from earlier layers to be used deeper in the network, while aiding in gradient flow and regularization.

InNetGAN: Inception Network-Based Generative Adversarial Network for Denoising Low-Dose Computed Tomography

Low-dose Computed Tomography (LDCT) has gained a great deal of attention in clinical procedures due to its ability to reduce the patient's risk of exposure to the X-ray radiation. However, reducing the X-ray dose increases the quantum noise and artifacts in the acquired LDCT images. As a result, it produces visually low-quality LDCT images that adversely affect the disease diagnosing and treatment planning in clinical procedures. Deep Learning (DL) has recently become the cutting-edge technology of LDCT denoising due to its high performance and data-driven execution compared to conventional denoising approaches. Although the DL-based models perform fairly well in LDCT noise reduction, some noise components are still retained in denoised LDCT images. One reason for this noise retention is the direct transmission of feature maps through the skip connections of contraction and extraction path-based DL modes. Therefore, in this study, we propose a Generative Adversarial Network with Inception network modules (InNetGAN) as a solution for filtering the noise transmission through skip connections and preserving the texture and fine structure of LDCT images. The proposed Generator is modeled based on the U-net architecture. The skip connections in the U-net architecture are modified with three different inception network modules to filter out the noise in the feature maps passing over them. The quantitative and qualitative experimental results have shown the performance of the InNetGAN model in reducing noise and preserving the subtle structures and texture details in LDCT images compared to the other state-of-the-art denoising algorithms.

Low-Dose CT Image Denoising with Improving WGAN and Hybrid Loss Function

The X-ray radiation from computed tomography (CT) brought us the potential risk. Simply decreasing the dose makes the CT images noisy and diagnostic performance compromised. Here, we develop a novel denoising low-dose CT image method. Our framework is based on an improved generative adversarial network coupling with the hybrid loss function, including the adversarial loss, perceptual loss, sharpness loss, and structural similarity loss. Among the loss function terms, perceptual loss and structural similarity loss are made use of to preserve textural details, and sharpness loss can make reconstruction images clear. The adversarial loss can sharp the boundary regions. The results of experiments show the proposed method can effectively remove noise and artifacts better than the state-of-the-art methods in the aspects of the visual effect, the quantitative measurements, and the texture details.

Efficient Computer-Aided Design of Dental Inlay Restoration: A Deep Adversarial Framework

Restoring the normal masticatory function of broken teeth is a challenging task primarily due to the defect location and size of a patient's teeth. In recent years, although some representative image-to-image transformation methods (e.g. Pix2Pix) can be potentially applicable to restore the missing crown surface, most of them fail to generate dental inlay surface with realistic crown details (e.g. occlusal groove) that are critical to the restoration of defective teeth with varying shapes. In this article, we design a computer-aided Deep Adversarial-driven dental Inlay reStoration (DAIS) framework to automatically reconstruct a realistic surface for a defective tooth. Specifically, DAIS consists of a Wasserstein generative adversarial network (WGAN) with a specially designed loss measurement, and a new local-global discriminator mechanism. The local discriminator focuses on missing regions to ensure the local consistency of a generated occlusal surface, while the global discriminator aims at defective teeth and adjacent teeth to assess if it is coherent as a whole. Experimental results demonstrate that DAIS is highly efficient to deal with a large area of missing teeth in arbitrary shapes and generate realistic occlusal surface completion. Moreover, the designed watertight inlay prostheses have enough anatomical morphology, thus providing higher clinical applicability compared with more state-of-the-art methods.

Weakly supervised anomaly segmentation in retinal OCT images using an adversarial learning approach

Lesion detection is a critical component of disease diagnosis, but the manual segmentation of lesions in medical images is time-consuming and experience-demanding. These issues have recently been addressed through deep learning models. However, most of the existing algorithms were developed using supervised training, which requires time-intensive manual labeling and prevents the model from detecting unaware lesions. As such, this study proposes a weakly supervised learning network based on CycleGAN for lesions segmentation in full-width optical coherence tomography (OCT) images. The model was trained to reconstruct underlying normal anatomic structures from abnormal input images, then the lesions can be detected by calculating the difference between the input and output images. A customized network architecture and a multi-scale similarity perceptual reconstruction loss were used to extend the CycleGAN model to transfer between objects exhibiting shape deformations. The proposed technique was validated using an open-source retinal OCT image dataset. Image-level anomaly detection and pixel-level lesion detection results were assessed using area-under-curve (AUC) and the Dice similarity coefficient, producing results of 96.94% and 0.8239, respectively, higher than all comparative methods. The average test time required to generate a single full-width image was 0.039 s, which is shorter than that reported in recent studies. These results indicate that our model can accurately detect and segment retinopathy lesions in real-time, without the need for supervised labeling. And we hope this method will be helpful to accelerate the clinical diagnosis process and reduce the misdiagnosis rate.

NIA-Network: Towards improving lung CT infection detection for COVID-19 diagnosis

During pandemics (e.g., COVID-19) physicians have to focus on diagnosing and treating patients, which often results in that only a limited amount of labeled CT images is available. Although recent semi-supervised learning algorithms may alleviate the problem of annotation scarcity, limited real-world CT images still cause those algorithms producing inaccurate detection results, especially in real-world COVID-19 cases. Existing models often cannot detect the small infected regions in COVID-19 CT images, such a challenge implicitly causes that many patients with minor symptoms are misdiagnosed and develop more severe symptoms, causing a higher mortality. In this paper, we propose a new method to address this challenge. Not only can we detect severe cases, but also detect minor symptoms using real-world COVID-19 CT images in which the source domain only includes limited labeled CT images but the target domain has a lot of unlabeled CT images. Specifically, we adopt Network-in-Network and Instance Normalization to build a new module (we term it NI module) and extract discriminative representations from CT images from both source and target domains. A domain classifier is utilized to implement infected region adaptation from source domain to target domain in an Adversarial Learning manner, and learns domain-invariant region proposal network (RPN) in the Faster R-CNN model. We call our model NIA-Network (Network-in-Network, Instance Normalization and Adversarial Learning), and conduct extensive experiments on two COVID-19 datasets to validate our approach. The experimental results show that our model can effectively detect infected regions with different sizes and achieve the highest diagnostic accuracy compared with existing SOTA methods.

Unraveling the interplay of image formation, data representation and learning in CT-based COPD phenotyping automation: The need for a meta-strategy

PURPOSE

In the literature on automated phenotyping of chronic obstructive pulmonary disease (COPD), there is a multitude of isolated classical machine learning and deep learning techniques, mostly investigating individual phenotypes, with small study cohorts and heterogeneous meta-parameters, e.g., different scan protocols or segmented regions. The objective is to compare the impact of different experimental setups, i.e., varying meta-parameters related to image formation and data representation, with the impact of the learning technique for subtyping automation for a variety of phenotypes. The identified associations of these parameters with automation performance and their interactions might be a first step towards a determination of optimal meta-parameters, i.e., a meta-strategy.

METHODS

A clinical cohort of 981 patients (53.8 ± 15.1 years, 554 male) was examined. The inspiratory CT images were analyzed to automate the diagnosis of 13 COPD phenotypes given by two radiologists. A benchmark feature set that integrates many quantitative criteria was extracted from the lung and trained a variety of learning algorithms on the first 654 patients (two thirds) and the respective algorithm retrospectively assessed the remaining 327 patients (one third). The automation performance was evaluated by the area under the receiver operating characteristic curve (AUC). 1717 experiments were conducted with varying meta-parameters such as reconstruction kernel, segmented regions and input dimensionality, i.e., number of extracted features. The association of the meta-parameters with the automation performance was analyzed by multivariable general linear model decomposition of the automation performance in the contributions of meta-parameters and the learning technique.

RESULTS

The automation performance varied strongly for varying meta-parameters. For emphysema-predominant phenotypes, an AUC of 93%-95% could be achieved for the best meta-configuration. The airways-predominant phenotypes led to a lower performance of 65%-85%, while smooth kernel configurations on average were unexpectedly superior to those with sharp kernels. The performance impact of meta-parameters, even that of often neglected ones like the missing-data imputation, was in general larger than that of the learning technique. Advanced learning techniques like 3D deep learning or automated machine learning yielded inferior automation performance for non-optimal meta-configurations in comparison to simple techniques with suitable meta-configurations. The best automation performance was achieved by a combination of modern learning techniques and a suitable meta-configuration.

CONCLUSIONS

Our results indicate that for COPD phenotype automation, study design parameters such as reconstruction kernel and the model input dimensionality should be adapted to the learning technique and may be more important than the technique itself. To achieve optimal automation and prediction results, the interaction between input those meta-parameters and the learning technique should be considered. This might be particularly relevant for the development of specific scan protocols for novel learning algorithms, and towards an understanding of good study design for automated phenotyping.

A review on Deep Learning approaches for low-dose Computed Tomography restoration

Computed Tomography (CT) is a widely use medical image modality in clinical medicine, because it produces excellent visualizations of fine structural details of the human body. In clinical procedures, it is desirable to acquire CT scans by minimizing the X-ray flux to prevent patients from being exposed to high radiation. However, these Low-Dose CT (LDCT) scanning protocols compromise the signal-to-noise ratio of the CT images because of noise and artifacts over the image space. Thus, various restoration methods have been published over the past 3 decades to produce high-quality CT images from these LDCT images. More recently, as opposed to conventional LDCT restoration methods, Deep Learning (DL)-based LDCT restoration approaches have been rather common due to their characteristics of being data-driven, high-performance, and fast execution. Thus, this study aims to elaborate on the role of DL techniques in LDCT restoration and critically review the applications of DL-based approaches for LDCT restoration. To achieve this aim, different aspects of DL-based LDCT restoration applications were analyzed. These include DL architectures, performance gains, functional requirements, and the diversity of objective functions. The outcome of the study highlights the existing limitations and future directions for DL-based LDCT restoration. To the best of our knowledge, there have been no previous reviews, which specifically address this topic.

A material decomposition method for dual-energy CT via dual interactive Wasserstein generative adversarial networks

PURPOSE

Dual-energy computed tomography (DECT) is highly promising for material characterization and identification, whereas reconstructed material-specific images are affected by magnified noise and beam-hardening artifacts. Although various DECT material decomposition methods have been proposed to solve this problem, the quality of the decomposed images is still unsatisfactory, particularly in the image edges. In this study, a data-driven approach using dual interactive Wasserstein generative adversarial networks (DIWGAN) is developed to improve DECT decomposition accuracy and perform edge-preserving images.

METHODS

In proposed DIWGAN, two interactive generators are used to synthesize decomposed images of two basis materials by modeling the spatial and spectral correlations from input DECT reconstructed images, and the corresponding discriminators are employed to distinguish the difference between the generated images and labels. The DECT images reconstructed from high- and low-energy bins are sent to two generators separately, and each generator synthesizes one material-specific image, thereby ensuring the specificity of the network modeling. In addition, the information from different energy bins is exploited through the feature sharing of two generators. During decomposition model training, a hybrid loss function including L₁ loss, edge loss, and adversarial loss is incorporated to preserve the texture and edges in the generated images. Additionally, a selector is employed to define the generator that should be trained in each iteration, which can ensure the modeling ability of two different generators and improve the material decomposition accuracy. The performance of the proposed method is evaluated using digital phantom, XCAT phantom, and real data from a mouse.

RESULTS

On the digital phantom, the regions of bone and soft tissue are strictly and accurately separated using the trained decomposition model. The material densities in different bone and soft-tissue regions are near the ground truth, and the error of material densities is lower than 3 mg/ml. The results from XCAT phantom show that the material-specific images generated by directed matrix inversion and iterative decomposition methods have severe noise and artifacts. Regarding to the learning-based methods, the decomposed images of fully convolutional network (FCN) and butterfly network (Butterfly-Net) still contain varying degrees of artifacts, while proposed DIWGAN can yield high quality images. Compared to Butterfly-Net, the root-mean-square error (RMSE) of soft-tissue images generated by the DIWGAN decreased by 0.01 g/ml, whereas the peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the soft-tissue images reached 31.43 dB and 0.9987, respectively. The mass densities of the decomposed materials are nearest to the ground truth when using the DIWGAN method. The noise standard deviation of the decomposition images reduced by 69%, 60%, 33%, and 21% compared with direct matrix inversion, iterative decomposition, FCN, and Butterfly-Net, respectively. Furthermore, the performance of the mouse data indicates the potential of the proposed material decomposition method in real scanned data.

CONCLUSIONS

A DECT material decomposition method based on deep learning is proposed, and the relationship between reconstructed and material-specific images is mapped by training the DIWGAN model. Results from both the simulation phantoms and real data demonstrate the advantages of this method in suppressing noise and beam-hardening artifacts.

Multi-Material Decomposition for Single Energy CT Using Material Sparsity Constraint

Multi-material decomposition (MMD) decomposes CT images into basis material images, and is a promising technique in clinical diagnostic CT to identify material compositions within the human body. MMD could be implemented on measurements obtained from spectral CT protocol, although spectral CT data acquisition is not readily available in most clinical environments. MMD methods using single energy CT (SECT), broadly applied in radiological departments of most hospitals, have been proposed in the literature while challenged by the inferior decomposition accuracy and the limited number of material bases due to the constrained material information in the SECT measurement. In this paper, we propose an image-domain SECT MMD method using material sparsity as an assistance under the condition that each voxel of the CT image contains at most two different elemental materials. L₀ norm represents the material sparsity constraint (MSC) and is integrated into the decomposition objective function with a least-square data fidelity term, total variation term, and a sum-to-one constraint of material volume fractions. An accelerated primal-dual (APD) algorithm with line-search scheme is applied to solve the problem. The pixelwise direct inversion method with the two-material assumption (TMA) is applied to estimate the initials. We validate the proposed method on phantom and patient data. Compared with the TMA method, the proposed MSC method increases the volume fraction accuracy (VFA) from 92.0% to 98.5% in the phantom study. In the patient study, the calcification area can be clearly visualized in the virtual non-contrast image generated by the proposed method, and has a similar shape to that in the ground-truth contrast-free CT image. The high decomposition image quality from the proposed method substantially facilitates the SECT-based MMD clinical applications.

Deep Generative Adversarial Networks: Applications in Musculoskeletal Imaging

In recent years, deep learning techniques have been applied in musculoskeletal radiology to increase the diagnostic potential of acquired images. Generative adversarial networks (GANs), which are deep neural networks that can generate or transform images, have the potential to aid in faster imaging by generating images with a high level of realism across multiple contrast and modalities from existing imaging protocols. This review introduces the key architectures of GANs as well as their technical background and challenges. Key research trends are highlighted, including: (a) reconstruction of high-resolution MRI; (b) image synthesis with different modalities and contrasts; (c) image enhancement that efficiently preserves high-frequency information suitable for human interpretation; (d) pixel-level segmentation with annotation sharing between domains; and (e) applications to different musculoskeletal anatomies. In addition, an overview is provided of the key issues wherein clinical applicability is challenging to capture with conventional performance metrics and expert evaluation. When clinically validated, GANs have the potential to improve musculoskeletal imaging. Keywords: Adults and Pediatrics, Computer Aided Diagnosis (CAD), Computer Applications-General (Informatics), Informatics, Skeletal-Appendicular, Skeletal-Axial, Soft Tissues/Skin © RSNA, 2021.

Realistic High-Resolution Body Computed Tomography Image Synthesis by Using Progressive Growing Generative Adversarial Network: Visual Turing Test

BACKGROUND

Generative adversarial network (GAN)-based synthetic images can be viable solutions to current supervised deep learning challenges. However, generating highly realistic images is a prerequisite for these approaches.

OBJECTIVE

The aim of this study was to investigate and validate the unsupervised synthesis of highly realistic body computed tomography (CT) images by using a progressive growing GAN (PGGAN) trained to learn the probability distribution of normal data.

METHODS

We trained the PGGAN by using 11,755 body CT scans. Ten radiologists (4 radiologists with <5 years of experience [Group I], 4 radiologists with 5-10 years of experience [Group II], and 2 radiologists with >10 years of experience [Group III]) evaluated the results in a binary approach by using an independent validation set of 300 images (150 real and 150 synthetic) to judge the authenticity of each image.

RESULTS

The mean accuracy of the 10 readers in the entire image set was higher than random guessing (1781/3000, 59.4% vs 1500/3000, 50.0%, respectively; P<.001). However, in terms of identifying synthetic images as fake, there was no significant difference in the specificity between the visual Turing test and random guessing (779/1500, 51.9% vs 750/1500, 50.0%, respectively; P=.29). The accuracy between the 3 reader groups with different experience levels was not significantly different (Group I, 696/1200, 58.0%; Group II, 726/1200, 60.5%; and Group III, 359/600, 59.8%; P=.36). Interreader agreements were poor (κ=0.11) for the entire image set. In subgroup analysis, the discrepancies between real and synthetic CT images occurred mainly in the thoracoabdominal junction and in the anatomical details.

CONCLUSIONS

The GAN can synthesize highly realistic high-resolution body CT images that are indistinguishable from real images; however, it has limitations in generating body images of the thoracoabdominal junction and lacks accuracy in the anatomical details.

Probabilistic self-learning framework for low-dose CT denoising

PURPOSE

Despite the indispensable role of x-ray computed tomography (CT) in diagnostic medicine, the associated harmful ionizing radiation dose is a major concern, as it may cause genetic diseases and cancer. Decreasing patients' exposure can reduce the radiation dose and hence the related risks, but it would inevitably induce higher quantum noise. Supervised deep learning techniques have been used to train deep neural networks for denoising low-dose CT (LDCT) images, but the success of such strategies requires massive sets of pixel-level paired LDCT and normal-dose CT (NDCT) images, which are rarely available in real clinical practice. Our purpose is to mitigate the data scarcity problem for deep learning-based LDCT denoising.

METHODS

To solve this problem, we devised a shift-invariant property-based neural network that uses only the LDCT images to characterize both the inherent pixel correlations and the noise distribution, shaping into our probabilistic self-learning (PSL) framework. The AAPM Low-dose CT Challenge dataset was used to train the network. Both simulated datasets and real dataset were employed to test the denoising performance as well as the model generalizability. The performance was compared to a conventional method (total variation (TV)-based), a popular self-learning method (noise2void (N2V)), and a well-known unsupervised learning method (CycleGAN) by using both qualitative visual inspection and quantitative metrics including peak signal-noise-ratio (PSNR), structural similarity index (SSIM) and contrast-to-noise-ratio (CNR). The standard deviations (STD) of selected flat regions were also calculated for comparison.

RESULTS

The PSL method can improve the averaged PSNR/SSIM values from 27.61/0.5939 (LDCT) to 30.50/0.6797. By contrast, the averaged PSNR/SSIM values were 31.49/0.7284 (TV), 29.43/0.6699 (N2V), and 29.79/0.6992 (CycleGAN). The averaged STDs of selected flat regions were calculated to be 132.3 HU (LDCT), 25.77 HU (TV), 19.95 HU (N2V), 75.06 HU (CycleGAN), 60.62 HU (PSL) and 57.28 HU (NDCT). As for the low-contrast lesion detectability quantification, the CNR were calculated to be 0.202 (LDCT), 0.356 (TV), 0.372 (N2V), 0.383 (CycleGAN), 0.399 (PSL), and 0.359 (NDCT). By visual inspection, we observed that the proposed PSL method can deliver a noise-suppressed and detail-preserved image, while the TV-based method would lead to the blocky artifact, the N2V method would produce over-smoothed structures and CT value biased effect, and the CycleGAN method would generate slightly noisy results with inaccurate CT values. We also verified the generalizability of the PSL method, which exhibited superior denoising performance among various testing datasets with different data distribution shifts.

CONCLUSIONS

A deep learning-based convolutional neural network can be trained without paired datasets. Qualitatively visual inspection showed the proposed PSL method can achieve superior denoising performance than all the competitors, despite that the employed quantitative metrics in terms of PSNR, SSIM and CNR did not always show consistently better values.

Application and Construction of Deep Learning Networks in Medical Imaging

Deep learning (DL) approaches are part of the machine learning (ML) subfield concerned with the development of computational models to train artificial intelligence systems. DL models are characterized by automatically extracting high-level features from the input data to learn the relationship between matching datasets. Thus, its implementation offers an advantage over common ML methods that often require the practitioner to have some domain knowledge of the input data to select the best latent representation. As a result of this advantage, DL has been successfully applied within the medical imaging field to address problems, such as disease classification and tumor segmentation for which it is difficult or impossible to determine which image features are relevant. Therefore, taking into consideration the positive impact of DL on the medical imaging field, this article reviews the key concepts associated with its evolution and implementation. The sections of this review summarize the milestones related to the development of the DL field, followed by a description of the elements of deep neural network and an overview of its application within the medical imaging field. Subsequently, the key steps necessary to implement a supervised DL application are defined, and associated limitations are discussed.

A review on medical imaging synthesis using deep learning and its clinical applications

This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs, and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were then summarized with discussion.

N2NSR-OCT: Simultaneous denoising and super-resolution in optical coherence tomography images using semisupervised deep learning

Optical coherence tomography (OCT) imaging shows a significant potential in clinical routines due to its noninvasive property. However, the quality of OCT images is generally limited by inherent speckle noise of OCT imaging and low sampling rate. To obtain high signal-to-noise ratio (SNR) and high-resolution (HR) OCT images within a short scanning time, we presented a learning-based method to recover high-quality OCT images from noisy and low-resolution OCT images. We proposed a semisupervised learning approach named N2NSR-OCT, to generate denoised and super-resolved OCT images simultaneously using up- and down-sampling networks (U-Net (Semi) and DBPN (Semi)). Additionally, two different super-resolution and denoising models with different upscale factors (2× and 4×) were trained to recover the high-quality OCT image of the corresponding down-sampling rates. The new semisupervised learning approach is able to achieve results comparable with those of supervised learning using up- and down-sampling networks, and can produce better performance than other related state-of-the-art methods in the aspects of maintaining subtle fine retinal structures.

CT artifact correction for sparse and truncated projection data using generative adversarial networks

PURPOSE

Computed tomography image reconstruction using truncated or sparsely acquired projection data to reduce radiation dose, iodine volume, and patient motion artifacts has been widely investigated. To continue these efforts, we investigated the use of machine learning-based reconstruction techniques using deep convolutional generative adversarial networks (DCGANs) and evaluated its effect using standard imaging metrics.

METHODS

Ten thousand head computed tomography (CT) scans were collected from the 2019 RSNA Intracranial Hemorrhage Detection and Classification Challenge dataset. Sinograms were simulated and then resampled in both a one-third truncated and one-third sparse manner. DCGANs were tasked with correcting the incomplete projection data, either in the sinogram domain where the full sinogram was recovered by the DCGAN and then reconstructed, or the reconstruction domain where the incomplete data were first reconstructed and the sparse or truncation artifacts were corrected by the DCGAN. Seventy-five hundred images were used for network training and 2500 were withheld for network assessment using mean absolute error (MAE), structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR) between results of different correction techniques. Image data from a quality-assurance phantom were also resampled in the two manners and corrected and reconstructed for network performance assessment using line profiles across high-contrast features, the modulation transfer function (MTF), noise power spectrum (NPS), and Hounsfield Unit (HU) linearity analysis.

RESULTS

Better agreement with the fully sampled reconstructions were achieved from sparse acquisition corrected in the sinogram domain and the truncated acquisition corrected in the reconstruction domain. MAE, SSIM, and PSNR showed quantitative improvement from the DCGAN correction techniques. HU linearity of the reconstructions was maintained by the correction techniques for the sparse and truncated acquisitions. MTF curves reached the 10% modulation cutoff frequency at 5.86 lp/cm for the truncated corrected reconstruction compared with 2.98 lp/cm for the truncated uncorrected reconstruction, and 5.36 lp/cm for the sparse corrected reconstruction compared with around 2.91 lp/cm for the sparse uncorrected reconstruction. NPS analyses yielded better agreement across a range of frequencies between the resampled corrected phantom and truth reconstructions.

CONCLUSIONS

We demonstrated the use of DCGANs for CT-image correction from sparse and truncated simulated projection data, while preserving imaging quality of the fully sampled projection data.

Spatial orthogonal attention generative adversarial network for MRI reconstruction

PURPOSE

Recent studies have witnessed that self-attention modules can better solve the vision understanding problems by capturing long-range dependencies. However, there are very few works designing a lightweight self-attention module to improve the quality of MRI reconstruction. Furthermore, it can be observed that several important self-attention modules (e.g., the non-local block) cause high computational complexity and need a huge number of GPU memory when the size of the input feature is large. The purpose of this study is to design a lightweight yet effective spatial orthogonal attention module (SOAM) to capture long-range dependencies, and develop a novel spatial orthogonal attention generative adversarial network, termed as SOGAN, to achieve more accurate MRI reconstruction.

METHODS

We first develop a lightweight SOAM, which can generate two small attention maps to effectively aggregate the long-range contextual information in vertical and horizontal directions, respectively. Then, we embed the proposed SOAMs into the concatenated convolutional autoencoders to form the generator of the proposed SOGAN.

RESULTS

The experimental results demonstrate that the proposed SOAMs improve the quality of the reconstructed MR images effectively by capturing long-range dependencies. Besides, compared with state-of-the-art deep learning-based CS-MRI methods, the proposed SOGAN reconstructs MR images more accurately, but with fewer model parameters.

CONCLUSIONS

The proposed SOAM is a lightweight yet effective self-attention module to capture long-range dependencies, thus, can improve the quality of MRI reconstruction to a large extent. Besides, with the help of SOAMs, the proposed SOGAN outperforms the state-of-the-art deep learning-based CS-MRI methods.

Noise reduction using novel loss functions to compute tissue mineral density and trabecular bone volume fraction on low resolution QCT

Micro-structural parameters of the thoracic or lumbar spine generally carry insufficient accuracy and precision for clinical in vivo studies when assessed on quantitative computed tomography (QCT). We propose a 3D convolutional neural network with specific loss functions for QCT noise reduction to compute micro-structural parameters such as tissue mineral density (TMD) and bone volume ratio (BV/TV) with significantly higher accuracy than using no or standard noise reduction filters. The vertebra-phantom study contained high resolution peripheral and clinical CT scans with simulated in vivo CT noise and nine repetitions of three different tube currents (100, 250 and 360 mAs). Five-fold cross validation was performed on 20466 purely spongy pairs of noisy and ground-truth patches. Comparison of training and test errors revealed high robustness against over-fitting. While not showing effects for the assessment of BMD and voxel-wise densities, the filter improved thoroughly the computation of TMD and BV/TV with respect to the unfiltered data. Root-mean-square and accuracy errors of low resolution TMD and BV/TV decreased to less than 17% of the initial values. Furthermore filtered low resolution scans revealed still more TMD- and BV/TV-relevant information than high resolution CT scans, either unfiltered or filtered with two state-of-the-art standard denoising methods. The proposed architecture is threshold and rotational invariant, applicable on a wide range of image resolutions at once, and likely serves for an accurate computation of further micro-structural parameters. Furthermore, it is less prone for over-fitting than neural networks that compute structural parameters directly. In conclusion, the method is potentially important for the diagnosis of osteoporosis and other bone diseases since it allows to assess relevant 3D micro-structural information from standard low exposure CT protocols such as 100 mAs and 120 kVp.

Artificial Intelligence Applications for Workflow, Process Optimization and Predictive Analytics

There is great potential for artificial intelligence (AI) applications, especially machine learning and natural language processing, in medical imaging. Much attention has been garnered by the image analysis tasks for diagnostic decision support and precision medicine, but there are many other potential applications of AI in radiology and have potential to enhance all levels of the radiology workflow and practice, including workflow optimization and support for interpretation tasks, quality and safety, and operational efficiency. This article reviews the important potential applications of informatics and AI related to process improvement and operations in the radiology department.

Edge-guided second-order total generalized variation for Gaussian noise removal from depth map

Total generalized variation models have recently demonstrated high-quality denoising capacity for single image. In this paper, we present an accurate denoising method for depth map. Our method uses a weighted second-order total generalized variational model for Gaussian noise removal. By fusing an edge indicator function into the regularization term of the second-order total generalized variational model to guide the diffusion of gradients, our method aims to use the first or second derivative to enhance the intensity of the diffusion tensor. We use the first-order primal–dual algorithm to minimize the proposed energy function and achieve high-quality denoising and edge preserving result for depth maps with high -intensity noise. Extensive quantitative and qualitative evaluations in comparison to bench-mark datasets show that the proposed method provides significant higher accuracy and visual improvements than many state-of-the-art denoising algorithms.

Creating Artificial Images for Radiology Applications Using Generative Adversarial Networks (GANs) - A Systematic Review

RATIONALE AND OBJECTIVES

Generative adversarial networks (GANs) are deep learning models aimed at generating fake realistic looking images. These novel models made a great impact on the computer vision field. Our study aims to review the literature on GANs applications in radiology.

MATERIALS AND METHODS

This systematic review followed the PRISMA guidelines. Electronic datasets were searched for studies describing applications of GANs in radiology. We included studies published up-to September 2019.

RESULTS

Data were extracted from 33 studies published between 2017 and 2019. Eighteen studies focused on CT images generation, ten on MRI, three on PET/MRI and PET/CT, one on ultrasound and one on X-ray. Applications in radiology included image reconstruction and denoising for dose and scan time reduction (fourteen studies), data augmentation (six studies), transfer between modalities (eight studies) and image segmentation (five studies). All studies reported that generated images improved the performance of the developed algorithms.

CONCLUSION

GANs are increasingly studied for various radiology applications. They enable the creation of new data, which can be used to improve clinical care, education and research.

Artificial intelligence in radiotherapy: a technological review

Radiation therapy (RT) is widely used to treat cancer. Technological advances in RT have occurred in the past 30 years. These advances, such as three-dimensional image guidance, intensity modulation, and robotics, created challenges and opportunities for the next breakthrough, in which artificial intelligence (AI) will possibly play important roles. AI will replace certain repetitive and labor-intensive tasks and improve the accuracy and consistency of others, particularly those with increased complexity because of technological advances. The improvement in efficiency and consistency is important to manage the increasing cancer patient burden to the society. Furthermore, AI may provide new functionalities that facilitate satisfactory RT. The functionalities include superior images for real-time intervention and adaptive and personalized RT. AI may effectively synthesize and analyze big data for such purposes. This review describes the RT workflow and identifies areas, including imaging, treatment planning, quality assurance, and outcome prediction, that benefit from AI. This review primarily focuses on deep-learning techniques, although conventional machine-learning techniques are also mentioned.

SACNN: Self-Attention Convolutional Neural Network for Low-Dose CT Denoising With Self-Supervised Perceptual Loss Network

Computed tomography (CT) is a widely used screening and diagnostic tool that allows clinicians to obtain a high-resolution, volumetric image of internal structures in a non-invasive manner. Increasingly, efforts have been made to improve the image quality of low-dose CT (LDCT) to reduce the cumulative radiation exposure of patients undergoing routine screening exams. The resurgence of deep learning has yielded a new approach for noise reduction by training a deep multi-layer convolutional neural networks (CNN) to map the low-dose to normal-dose CT images. However, CNN-based methods heavily rely on convolutional kernels, which use fixed-size filters to process one local neighborhood within the receptive field at a time. As a result, they are not efficient at retrieving structural information across large regions. In this paper, we propose a novel 3D self-attention convolutional neural network for the LDCT denoising problem. Our 3D self-attention module leverages the 3D volume of CT images to capture a wide range of spatial information both within CT slices and between CT slices. With the help of the 3D self-attention module, CNNs are able to leverage pixels with stronger relationships regardless of their distance and achieve better denoising results. In addition, we propose a self-supervised learning scheme to train a domain-specific autoencoder as the perceptual loss function. We combine these two methods and demonstrate their effectiveness on both CNN-based neural networks and WGAN-based neural networks with comprehensive experiments. Tested on the AAPM-Mayo Clinic Low Dose CT Grand Challenge data set, our experiments demonstrate that self-attention (SA) module and autoencoder (AE) perceptual loss function can efficiently enhance traditional CNNs and can achieve comparable or better results than the state-of-the-art methods.

A Task-dependent Investigation on Dose and Texture in CT Image Reconstruction

Localizing and characterizing clinically-significant lung nodules, a potential precursor to lung cancer, at the lowest achievable radiation dose is demanded to minimize the stochastic radiation effects from x-ray computed tomography (CT). A minimal dose level is heavily dependent on the image reconstruction algorithms and clinical task, in which the tissue texture always plays an important role. This study aims to investigate the dependence through a task-based evaluation at multiple dose levels and variable textures in reconstructions with prospective patient studies. 133 patients with a suspicious pulmonary nodule scheduled for biopsy were recruited and the data was acquired at120kVp with three different dose levels of 100, 40 and 20mAs. Three reconstruction algorithms were implemented: analytical filtered back-projection (FBP) with optimal noise filtering; statistical Markov random field (MRF) model with optimal Huber weighting (MRF-H) for piecewise smooth reconstruction; and tissue-specific texture model (MRF-T) for texture preserved statistical reconstruction. Experienced thoracic radiologists reviewed and scored all images at random, blind to the CT dose and reconstruction algorithms. The radiologists identified the nodules in each image including the 133 biopsy target nodules and 66 other non-target nodules. For target nodule characterization, only MRF-T at 40mAs showed no statistically significant difference from FBP at 100mAs. For localizing both the target nodules and the non-target nodules, some as small as 3mm, MRF-T at 40 and 20mAs levels showed no statistically significant difference from FBP at 100mAs, respectively. MRF-H and FBP at 40 and 20mAs levels performed statistically differently from FBP at 100mAs. This investigation concluded that (1) the textures in the MRF-T reconstructions improves both the tasks of localizing and characterizing nodules at low dose CT and (2) the task of characterizing nodules is more challenging than the task of localizing nodules and needs more dose or enhanced textures from reconstruction.

Quadratic Autoencoder (Q-AE) for Low-Dose CT Denoising

Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power of quadratic neurons in popular network architectures, simulating human-like learning in the form of "quadratic-neuron-based deep learning". Our prior theoretical studies have shown important merits of quadratic neurons and networks in representation, efficiency, and interpretability. In this paper, we use quadratic neurons to construct an encoder-decoder structure, referred as the quadratic autoencoder, and apply it to low-dose CT denoising. The experimental results on the Mayo low-dose CT dataset demonstrate the utility and robustness of quadratic autoencoder in terms of image denoising and model efficiency. To our best knowledge, this is the first time that the deep learning approach is implemented with a new type of neurons and demonstrates a significant potential in the medical imaging field.

MRI Cross-Modality Image-to-Image Translation

We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images. Our proposed method performs Image Modality Translation (abbreviated as IMT) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity in brain structures. Our framework can serve as an auxiliary method in medical use and has great application potential. Based on our proposed framework, we first propose a method for cross-modality registration by fusing the deformation fields to adopt the cross-modality information from translated modalities. Second, we propose an approach for MRI segmentation, translated multichannel segmentation (TMS), where given modalities, along with translated modalities, are segmented by fully convolutional networks (FCN) in a multichannel manner. Both of these two methods successfully adopt the cross-modality information to improve the performance without adding any extra data. Experiments demonstrate that our proposed framework advances the state-of-the-art on five brain MRI datasets. We also observe encouraging results in cross-modality registration and segmentation on some widely adopted brain datasets. Overall, our work can serve as an auxiliary method in medical use and be applied to various tasks in medical fields.

Deep Learning Based High-Resolution Reconstruction of Trabecular Bone Microstructures from Low-Resolution CT Scans using GAN-CIRCLE

Osteoporosis is a common age-related disease characterized by reduced bone density and increased fracture-risk. Microstructural quality of trabecular bone (Tb), commonly found at axial skeletal sites and at the end of long bones, is an important determinant of bone-strength and fracture-risk. High-resolution emerging CT scanners enable in vivo measurement of Tb microstructures at peripheral sites. However, resolution-dependence of microstructural measures and wide resolution-discrepancies among various CT scanners together with rapid upgrades in technology warrant data harmonization in CT-based cross-sectional and longitudinal bone studies. This paper presents a deep learning-based method for high-resolution reconstruction of Tb microstructures from low-resolution CT scans using GAN-CIRCLE. A network was developed and evaluated using post-registered ankle CT scans of nineteen volunteers on both low- and high-resolution CT scanners. 9,000 matching pairs of low- and high-resolution patches of size 64×64 were randomly harvested from ten volunteers for training and validation. Another 5,000 matching pairs of patches from nine other volunteers were used for evaluation. Quantitative comparison shows that predicted high-resolution scans have significantly improved structural similarity index (p < 0.01) with true high-resolution scans as compared to the same metric for low-resolution data. Different Tb microstructural measures such as thickness, spacing, and network area density are also computed from low- and predicted high-resolution images, and compared with the values derived from true high-resolution scans. Thickness and network area measures from predicted images showed higher agreement with true high-resolution CT (CCC = [0.95, 0.91]) derived values than the same measures from low-resolution images (CCC = [0.72, 0.88]).

Noise reduction in optical coherence tomography images using a deep neural network with perceptually-sensitive loss function

Optical coherence tomography (OCT) is susceptible to the coherent noise, which is the speckle noise that deteriorates contrast and the detail structural information of OCT images, thus imposing significant limitations on the diagnostic capability of OCT. In this paper, we propose a novel OCT image denoising method by using an end-to-end deep learning network with a perceptually-sensitive loss function. The method has been validated on OCT images acquired from healthy volunteers' eyes. The label images for training and evaluating OCT denoising deep learning models are images generated by averaging 50 frames of respective registered B-scans acquired from a region with scans occurring in one direction. The results showed that the new approach can outperform other related denoising methods on the aspects of preserving detail structure information of retinal layers and improving the perceptual metrics in the human visual perception.

[Improvement in Image Quality of CBCT during Treatment by Cycle Generative Adversarial Network]

PURPOSE

Volumetric modulated arc therapy (VMAT) can acquire projection images during rotational irradiation, and cone-beam computed tomography (CBCT) images during VMAT delivery can be reconstructed. The poor quality of CBCT images prevents accurate recognition of organ position during the treatment. The purpose of this study was to improve the image quality of CBCT during the treatment by cycle generative adversarial network (CycleGAN).

METHOD

Twenty patients with clinically localized prostate cancer were treated with VMAT, and projection images for intra-treatment CBCT (iCBCT) were acquired. Synthesis of PCT (SynPCT) with improved image quality by CycleGAN requires only unpaired and unaligned iCBCT and planning CT (PCT) images for training. We performed visual and quantitative evaluation to compare iCBCT, SynPCT and PCT deformable image registration (DIR) to confirm the clinical usefulness.

RESULT

We demonstrated suitable CycleGAN networks and hyperparameters for SynPCT. The image quality of SynPCT improved visually and quantitatively while preserving anatomical structures of the original iCBCT. The undesirable deformation of PCT was reduced when SynPCT was used as its reference instead of iCBCT.

CONCLUSION

We have performed image synthesis with preservation of organ position by CycleGAN for iCBCT and confirmed the clinical usefulness.

CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE)

In this paper, we present a semi-supervised deep learning approach to accurately recover high-resolution (HR) CT images from low-resolution (LR) counterparts. Specifically, with the generative adversarial network (GAN) as the building block, we enforce the cycle-consistency in terms of the Wasserstein distance to establish a nonlinear end-to-end mapping from noisy LR input images to denoised and deblurred HR outputs. We also include the joint constraints in the loss function to facilitate structural preservation. In this process, we incorporate deep convolutional neural network (CNN), residual learning, and network in network techniques for feature extraction and restoration. In contrast to the current trend of increasing network depth and complexity to boost the imaging performance, we apply a parallel 1×1 CNN to compress the output of the hidden layer and optimize the number of layers and the number of filters for each convolutional layer. The quantitative and qualitative evaluative results demonstrate that our proposed model is accurate, efficient and robust for super-resolution (SR) image restoration from noisy LR input images. In particular, we validate our composite SR networks on three large-scale CT datasets, and obtain promising results as compared to the other state-of-the-art methods.