Medical Image Synthesis with Deep Convolutional Adversarial Networks

Medical image analysis based on deep learning approach

Medical imaging plays a significant role in different clinical applications such as medical procedures used for early detection, monitoring, diagnosis, and treatment evaluation of various medical conditions. Basicsof the principles and implementations of artificial neural networks and deep learning are essential for understanding medical image analysis in computer vision. Deep Learning Approach (DLA) in medical image analysis emerges as a fast-growing research field. DLA has been widely used in medical imaging to detect the presence or absence of the disease. This paper presents the development of artificial neural networks, comprehensive analysis of DLA, which delivers promising medical imaging applications. Most of the DLA implementations concentrate on the X-ray images, computerized tomography, mammography images, and digital histopathology images. It provides a systematic review of the articles for classification, detection, and segmentation of medical images based on DLA. This review guides the researchers to think of appropriate changes in medical image analysis based on DLA.

Applying deep learning in digital breast tomosynthesis for automatic breast cancer detection: A review

The relatively recent reintroduction of deep learning has been a revolutionary force in the interpretation of diagnostic imaging studies. However, the technology used to acquire those images is undergoing a revolution itself at the very same time. Digital breast tomosynthesis (DBT) is one such technology, which has transformed the field of breast imaging. DBT, a form of three-dimensional mammography, is rapidly replacing the traditional two-dimensional mammograms. These parallel developments in both the acquisition and interpretation of breast images present a unique case study in how modern AI systems can be designed to adapt to new imaging methods. They also present a unique opportunity for co-development of both technologies that can better improve the validity of results and patient outcomes. In this review, we explore the ways in which deep learning can be best integrated into breast cancer screening workflows using DBT. We first explain the principles behind DBT itself and why it has become the gold standard in breast screening. We then survey the foundations of deep learning methods in diagnostic imaging, and review the current state of research into AI-based DBT interpretation. Finally, we present some of the limitations of integrating AI into clinical practice and the opportunities these present in this burgeoning field.

Assessment of Generative Adversarial Networks for Synthetic Anterior Segment Optical Coherence Tomography Images in Closed-Angle Detection

PURPOSE

To develop generative adversarial networks (GANs) that synthesize realistic anterior segment optical coherence tomography (AS-OCT) images and evaluate deep learning (DL) models that are trained on real and synthetic datasets for detecting angle closure.

METHODS

The GAN architecture was adopted and trained on the dataset with AS-OCT images collected from the Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, synthesizing open- and closed-angle AS-OCT images. A visual Turing test with two glaucoma specialists was performed to assess the image quality of real and synthetic images. DL models, trained on either real or synthetic datasets, were developed. Using the clinicians' grading of the AS-OCT images as the reference standard, we compared the diagnostic performance of open-angle vs. closed-angle detection of DL models and the AS-OCT parameter, defined as a trabecular-iris space area 750 µm anterior to the scleral spur (TISA750), in a small independent validation dataset.

RESULTS

The GAN training included 28,643 AS-OCT anterior chamber angle (ACA) images. The real and synthetic datasets for DL model training have an equal distribution of open- and closed-angle images (all with 10,000 images each). The independent validation dataset included 238 open-angle and 243 closed-angle AS-OCT ACA images. The image quality of real versus synthetic AS-OCT images was similar, as assessed by the two glaucoma specialists, except for the scleral spur visibility. For the independent validation dataset, both DL models achieved higher areas under the curve compared with TISA750. Two DL models had areas under the curve of 0.97 (95% confidence interval, 0.96-0.99) and 0.94 (95% confidence interval, 0.92-0.96).

CONCLUSIONS

The GAN synthetic AS-OCT images appeared to be of good quality, according to the glaucoma specialists. The DL models, trained on all-synthetic AS-OCT images, can achieve high diagnostic performance.

TRANSLATIONAL RELEVANCE

The GANs can generate realistic AS-OCT images, which can also be used to train DL models.

Oncoimmunology Meets Organs-on-Chip

Oncoimmunology represents a biomedical research discipline coined to study the roles of immune system in cancer progression with the aim of discovering novel strategies to arm it against the malignancy. Infiltration of immune cells within the tumor microenvironment is an early event that results in the establishment of a dynamic cross-talk. Here, immune cells sense antigenic cues to mount a specific anti-tumor response while cancer cells emanate inhibitory signals to dampen it. Animals models have led to giant steps in this research context, and several tools to investigate the effect of immune infiltration in the tumor microenvironment are currently available. However, the use of animals represents a challenge due to ethical issues and long duration of experiments. Organs-on-chip are innovative tools not only to study how cells derived from different organs interact with each other, but also to investigate on the crosstalk between immune cells and different types of cancer cells. In this review, we describe the state-of-the-art of microfluidics and the impact of OOC in the field of oncoimmunology underlining the importance of this system in the advancements on the complexity of tumor microenvironment.

Deep-Learning Generated Synthetic Double Inversion Recovery Images Improve Multiple Sclerosis Lesion Detection

OBJECTIVES

The aim of the study was to implement a deep-learning tool to produce synthetic double inversion recovery (synthDIR) images and compare their diagnostic performance to conventional sequences in patients with multiple sclerosis (MS).

MATERIALS AND METHODS

For this retrospective analysis, 100 MS patients (65 female, 37 [22-68] years) were randomly selected from a prospective observational cohort between 2014 and 2016. In a subset of 50 patients, an artificial neural network (DiamondGAN) was trained to generate a synthetic DIR (synthDIR) from standard acquisitions (T1, T2, and fluid-attenuated inversion recovery [FLAIR]). With the resulting network, synthDIR was generated for the remaining 50 subjects. These images as well as conventionally acquired DIR (trueDIR) and FLAIR images were assessed for MS lesions by 2 independent readers, blinded to the source of the DIR image. Lesion counts in the different modalities were compared using a Wilcoxon signed-rank test, and interrater analysis was performed. Contrast-to-noise ratios were compared for objective image quality.

RESULTS

Utilization of synthDIR allowed to detect significantly more lesions compared with the use of FLAIR images (31.4 ± 20.7 vs 22.8 ± 12.7, P < 0.001). This improvement was mainly attributable to an improved depiction of juxtacortical lesions (12.3 ± 10.8 vs 7.2 ± 5.6, P < 0.001). Interrater reliability was excellent in FLAIR 0.92 (95% confidence interval [CI], 0.85-0.95), synthDIR 0.93 (95% CI, 0.87-0.96), and trueDIR 0.95 (95% CI, 0.85-0.98).Contrast-to-noise ratio in synthDIR exceeded that of FLAIR (22.0 ± 6.4 vs 16.7 ± 3.6, P = 0.009); no significant difference was seen in comparison to trueDIR (22.0 ± 6.4 vs 22.4 ± 7.9, P = 0.87).

CONCLUSIONS

Computationally generated DIR images improve lesion depiction compared with the use of standard modalities. This method demonstrates how artificial intelligence can help improving imaging in specific pathologies.

Computer-Aided Diagnosis of COVID-19 CT Scans Based on Spatiotemporal Information Fusion

Coronavirus disease (COVID-19) is highly contagious and pathogenic. Currently, the diagnosis of COVID-19 is based on nucleic acid testing, but it has false negatives and hysteresis. The use of lung CT scans can help screen and effectively monitor diagnosed cases. The application of computer-aided diagnosis technology can reduce the burden on doctors, which is conducive to rapid and large-scale diagnostic screening. In this paper, we proposed an automatic detection method for COVID-19 based on spatiotemporal information fusion. Using the segmentation network in the deep learning method to segment the lung area and the lesion area, the spatiotemporal information features of multiple CT scans are extracted to perform auxiliary diagnosis analysis. The performance of this method was verified on the collected dataset. We achieved the classification of COVID-19 CT scans and non-COVID-19 CT scans and analyzed the development of the patients' condition through the CT scans. The average accuracy rate is 96.7%, sensitivity is 95.2%, and F1 score is 95.9%. Each scan takes about 30 seconds for detection.

DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis

Cycle-consistent generative adversarial network (CycleGAN) has been widely used for cross-domain medical image synthesis tasks particularly due to its ability to deal with unpaired data. However, most CycleGAN-based synthesis methods cannot achieve good alignment between the synthesized images and data from the source domain, even with additional image alignment losses. This is because the CycleGAN generator network can encode the relative deformations and noises associated to different domains. This can be detrimental for the downstream applications that rely on the synthesized images, such as generating pseudo-CT for PET-MR attenuation correction. In this paper, we present a deformation invariant cycle-consistency model that can filter out these domain-specific deformation. The deformation is globally parameterized by thin-plate-spline (TPS), and locally learned by modified deformable convolutional layers. Robustness to domain-specific deformations has been evaluated through experiments on multi-sequence brain MR data and multi-modality abdominal CT and MR data. Experiment results demonstrated that our method can achieve better alignment between the source and target data while maintaining superior image quality of signal compared to several state-of-the-art CycleGAN-based methods.

T1-weighted and T2-weighted MRI image synthesis with convolutional generative adversarial networks

Background

The objective of this study was to propose an optimal input image quality for a conditional generative adversarial network (GAN) in T1-weighted and T2-weighted magnetic resonance imaging (MRI) images.

Materials and methods

A total of 2,024 images scanned from 2017 to 2018 in 104 patients were used. The prediction framework of T1-weighted to T2-weighted MRI images and T2-weighted to T1-weighted MRI images were created with GAN. Two image sizes (512 × 512 and 256 × 256) and two grayscale level conversion method (simple and adaptive) were used for the input images. The images were converted from 16-bit to 8-bit by dividing with 256 levels in a simple conversion method. For the adaptive conversion method, the unused levels were eliminated in 16-bit images, which were converted to 8-bit images by dividing with the value obtained after dividing the maximum pixel value with 256.

Results

The relative mean absolute error (rMAE ) was 0.15 for T1-weighted to T2-weighted MRI images and 0.17 for T2-weighted to T1-weighted MRI images with an adaptive conversion method, which was the smallest. Moreover, the adaptive conversion method has a smallest mean square error (rMSE) and root mean square error (rRMSE), and the largest peak signal-to-noise ratio (PSNR) and mutual information (MI). The computation time depended on the image size.

Conclusions

Input resolution and image size affect the accuracy of prediction. The proposed model and approach of prediction framework can help improve the versatility and quality of multi-contrast MRI tests without the need for prolonged examinations.

Residual cyclegan for robust domain transformation of histopathological tissue slides

Variation between stains in histopathology is commonplace across different medical centers. This can have a significant effect on the reliability of machine learning algorithms. In this paper, we propose to reduce performance variability by using -consistent generative adversarial (CycleGAN) networks to remove staining variation. We improve upon the regular CycleGAN by incorporating residual learning. We comprehensively evaluate the performance of our stain transformation method and compare its usefulness in addition to extensive data augmentation to enhance the robustness of tissue segmentation algorithms. Our steps are as follows: first, we train a model to perform segmentation on tissue slides from a single source center, while heavily applying augmentations to increase robustness to unseen data. Second, we evaluate and compare the segmentation performance on data from other centers, both with and without applying our CycleGAN stain transformation. We compare segmentation performances in a colon tissue segmentation and kidney tissue segmentation task, covering data from 6 different centers. We show that our transformation method improves the overall Dice coefficient by 9% over the non-normalized target data and by 4% over traditional stain transformation in our colon tissue segmentation task. For kidney segmentation, our residual CycleGAN increases performance by 10% over no transformation and around 2% compared to the non-residual CycleGAN.

mustGAN: multi-stream Generative Adversarial Networks for MR Image Synthesis

Multi-contrast MRI protocols increase the level of morphological information available for diagnosis. Yet, the number and quality of contrasts are limited in practice by various factors including scan time and patient motion. Synthesis of missing or corrupted contrasts from other high-quality ones can alleviate this limitation. When a single target contrast is of interest, common approaches for multi-contrast MRI involve either one-to-one or many-to-one synthesis methods depending on their input. One-to-one methods take as input a single source contrast, and they learn a latent representation sensitive to unique features of the source. Meanwhile, many-to-one methods receive multiple distinct sources, and they learn a shared latent representation more sensitive to common features across sources. For enhanced image synthesis, we propose a multi-stream approach that aggregates information across multiple source images via a mixture of multiple one-to-one streams and a joint many-to-one stream. The complementary feature maps generated in the one-to-one streams and the shared feature maps generated in the many-to-one stream are combined with a fusion block. The location of the fusion block is adaptively modified to maximize task-specific performance. Quantitative and radiological assessments on T₁,- T₂-, PD-weighted, and FLAIR images clearly demonstrate the superior performance of the proposed method compared to previous state-of-the-art one-to-one and many-to-one methods.

Automated Segmentation Method for Low Field 3D Stomach MRI Using Transferred Learning Image Enhancement Network

Accurate segmentation of abdominal organs has always been a difficult problem, especially for organs with cavities. And MRI-guided radiotherapy is particularly attractive for abdominal targets compared with low CT contrast. But in the limit of radiotherapy environment, only low field MRI segmentation can be used for stomach location, tracking, and treatment planning. In clinical applications, the existing 3D segmentation network model is trained by the low field MRI, and the segmentation result cannot be used in radiotherapy plan since the bad segmentation performance. Another way is that historical high field intensity MR images are directly used for data expansion to network learning; there will be a domain shift problem. How to use different domain images to improve the segmentation accuracy of deep neural network? A 3D low field MRI stomach segmentation method based on transfer learning image enhancement is proposed in this paper. In this method, Cycle Generative Adversarial Network (CycleGAN) is used to construct and learn the mapping relationship between high and low field intensity MRI and to overcome domain shift. Then, the image generated by the high field intensity MRI through the CycleGAN network is with transferred information as the extended data. The low field MRI combines these extended datasets to form the training data for training the 3D Res-Unet segmentation network. Furthermore, the convolution layer, batch normalization layer, and Relu layer together were replaced with a residual module to relieve the gradient disappearance of the neural network. The experimental results show that the Dice coefficient is 2.5 percent better than the baseline method. The over segmentation and under segmentation are reduced by 0.7 and 5.5 percent, respectively. And the sensitivity is improved by 6.4 percent.

CMC: A consensus multi-view clustering model for predicting Alzheimer's disease progression

Machine learning has been used in the past for the auxiliary diagnosis of Alzheimer's Disease (AD). However, most existing technologies only explore single-view data, require manual parameter setting and focus on two-class (i.e., dementia or not) classification problems. Unlike single-view data, multi-view data provide more powerful feature representation capability. Learning with multi-view data is referred to as multi-view learning, which has received certain attention in recent years. In this paper, we propose a new multi-view clustering model called Consensus Multi-view Clustering (CMC) based on nonnegative matrix factorization for predicting the multiple stages of AD progression. The proposed CMC performs multi-view learning idea to fully capture data features with limited medical images, approaches similarity relations between different entities, addresses the shortcoming from multi-view fusion that requires manual setting parameters, and further acquires a consensus representation containing shared features and complementary knowledge of multiple view data. It not only can improve the predication performance of AD, but also can screen and classify the symptoms of different AD's phases. Experimental results using data with twelve views constructed by brain Magnetic Resonance Imaging (MRI) database from Alzheimer's Disease Neuroimaging Initiative expound and prove the effectiveness of the proposed model.

LCPR-Net: low-count PET image reconstruction using the domain transform and cycle-consistent generative adversarial networks

BACKGROUND

Reducing the radiation tracer dose and scanning time during positron emission tomography (PET) imaging can reduce the cost of the tracer, reduce motion artifacts, and increase the efficiency of the scanner. However, the reconstructed images to be noisy. It is very important to reconstruct high-quality images with low-count (LC) data. Therefore, we propose a deep learning method called LCPR-Net, which is used for directly reconstructing full-count (FC) PET images from corresponding LC sinogram data.

METHODS

Based on the framework of a generative adversarial network (GAN), we enforce a cyclic consistency constraint on the least-squares loss to establish a nonlinear end-to-end mapping process from LC sinograms to FC images. In this process, we merge a convolutional neural network (CNN) and a residual network for feature extraction and image reconstruction. In addition, the domain transform (DT) operation sends a priori information to the cycle-consistent GAN (CycleGAN) network, avoiding the need for a large amount of computational resources to learn this transformation.

RESULTS

The main advantages of this method are as follows. First, the network can use LC sinogram data as input to directly reconstruct an FC PET image. The reconstruction speed is faster than that provided by model-based iterative reconstruction. Second, reconstruction based on the CycleGAN framework improves the quality of the reconstructed image.

CONCLUSIONS

Compared with other state-of-the-art methods, the quantitative and qualitative evaluation results show that the proposed method is accurate and effective for FC PET image reconstruction.

Deep Learning for Multigrade Brain Tumor Classification in Smart Healthcare Systems: A Prospective Survey

Brain tumor is one of the most dangerous cancers in people of all ages, and its grade recognition is a challenging problem for radiologists in health monitoring and automated diagnosis. Recently, numerous methods based on deep learning have been presented in the literature for brain tumor classification (BTC) in order to assist radiologists for a better diagnostic analysis. In this overview, we present an in-depth review of the surveys published so far and recent deep learning-based methods for BTC. Our survey covers the main steps of deep learning-based BTC methods, including preprocessing, features extraction, and classification, along with their achievements and limitations. We also investigate the state-of-the-art convolutional neural network models for BTC by performing extensive experiments using transfer learning with and without data augmentation. Furthermore, this overview describes available benchmark data sets used for the evaluation of BTC. Finally, this survey does not only look into the past literature on the topic but also steps on it to delve into the future of this area and enumerates some research directions that should be followed in the future, especially for personalized and smart healthcare.

MAGAN: Mask Attention Generative Adversarial Network for Liver Tumor CT Image Synthesis

For deep learning, the size of the dataset greatly affects the final training effect. However, in the field of computer-aided diagnosis, medical image datasets are often limited and even scarce. We aim to synthesize medical images and enlarge the size of the medical image dataset. In the present study, we synthesized the liver CT images with a tumor based on the mask attention generative adversarial network (MAGAN). We masked the pixels of the liver tumor in the image as the attention map. And both the original image and attention map were loaded into the generator network to obtain the synthesized images. Then, the original images, the attention map, and the synthesized images were all loaded into the discriminator network to determine if the synthesized images were real or fake. Finally, we can use the generator network to synthesize liver CT images with a tumor. The experiments showed that our method outperformed the other state-of-the-art methods and can achieve a mean peak signal-to-noise ratio (PSNR) of 64.72 dB. All these results indicated that our method can synthesize liver CT images with a tumor and build a large medical image dataset, which may facilitate the progress of medical image analysis and computer-aided diagnosis. An earlier version of our study has been presented as a preprint in the following link: https://www.researchsquare.com/article/rs-41685/v1.

Anatomy-Regularized Representation Learning for Cross-Modality Medical Image Segmentation

An increasing number of studies are leveraging unsupervised cross-modality synthesis to mitigate the limited label problem in training medical image segmentation models. They typically transfer ground truth annotations from a label-rich imaging modality to a label-lacking imaging modality, under an assumption that different modalities share the same anatomical structure information. However, since these methods commonly use voxel/pixel-wise cycle-consistency to regularize the mappings between modalities, high-level semantic information is not necessarily preserved. In this paper, we propose a novel anatomy-regularized representation learning approach for segmentation-oriented cross-modality image synthesis. It learns a common feature encoding across different modalities to form a shared latent space, where 1) the input and its synthesis present consistent anatomical structure information, and 2) the transformation between two images in one domain is preserved by their syntheses in another domain. We applied our method to the tasks of cross-modality skull segmentation and cardiac substructure segmentation. Experimental results demonstrate the superiority of our method in comparison with state-of-the-art cross-modality medical image segmentation methods.

A review on medical imaging synthesis using deep learning and its clinical applications

This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs, and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were then summarized with discussion.

Deep learning-based real-time volumetric imaging for lung stereotactic body radiation therapy: a proof of concept study

Due to the inter- and intra- variation of respiratory motion, it is highly desired to provide real-time volumetric images during the treatment delivery of lung stereotactic body radiation therapy (SBRT) for accurate and active motion management. In this proof-of-concept study, we propose a novel generative adversarial network integrated with perceptual supervision to derive instantaneous volumetric images from a single 2D projection. Our proposed network, named TransNet, consists of three modules, i.e. encoding, transformation and decoding modules. Rather than only using image distance loss between the generated 3D images and the ground truth 3D CT images to supervise the network, perceptual loss in feature space is integrated into loss function to force the TransNet to yield accurate lung boundary. Adversarial supervision is also used to improve the realism of generated 3D images. We conducted a simulation study on 20 patient cases, who had received lung SBRT treatments in our institution and undergone 4D-CT simulation, and evaluated the efficacy and robustness of our method for four different projection angles, i.e. 0°, 30°, 60° and 90°. For each 3D CT image set of a breathing phase, we simulated its 2D projections at these angles. For each projection angle, a patient's 3D CT images of 9 phases and the corresponding 2D projection data were used to train our network for that specific patient, with the remaining phase used for testing. The mean absolute error of the 3D images obtained by our method are 99.3 ± 14.1 HU. The peak signal-to-noise ratio and structural similarity index metric within the tumor region of interest are 15.4 ± 2.5 dB and 0.839 ± 0.090, respectively. The center of mass distance between the manual tumor contours on the 3D images obtained by our method and the manual tumor contours on the corresponding 3D phase CT images are within 2.6 mm, with a mean value of 1.26 mm averaged over all the cases. Our method has also been validated in a simulated challenging scenario with increased respiratory motion amplitude and tumor shrinkage, and achieved acceptable results. Our experimental results demonstrate the feasibility and efficacy of our 2D-to-3D method for lung cancer patients, which provides a potential solution for in-treatment real-time on-board volumetric imaging for tumor tracking and dose delivery verification to ensure the effectiveness of lung SBRT treatment.

Deep learning for brain disorders: from data processing to disease treatment

In order to reach precision medicine and improve patients' quality of life, machine learning is increasingly used in medicine. Brain disorders are often complex and heterogeneous, and several modalities such as demographic, clinical, imaging, genetics and environmental data have been studied to improve their understanding. Deep learning, a subpart of machine learning, provides complex algorithms that can learn from such various data. It has become state of the art in numerous fields, including computer vision and natural language processing, and is also growingly applied in medicine. In this article, we review the use of deep learning for brain disorders. More specifically, we identify the main applications, the concerned disorders and the types of architectures and data used. Finally, we provide guidelines to bridge the gap between research studies and clinical routine.

Unsupervised MR-to-CT Synthesis Using Structure-Constrained CycleGAN

Synthesizing a CT image from an available MR image has recently emerged as a key goal in radiotherapy treatment planning for cancer patients. CycleGANs have achieved promising results on unsupervised MR-to-CT image synthesis; however, because they have no direct constraints between input and synthetic images, cycleGANs do not guarantee structural consistency between these two images. This means that anatomical geometry can be shifted in the synthetic CT images, clearly a highly undesirable outcome in the given application. In this paper, we propose a structure-constrained cycleGAN for unsupervised MR-to-CT synthesis by defining an extra structure-consistency loss based on the modality independent neighborhood descriptor. We also utilize a spectral normalization technique to stabilize the training process and a self-attention module to model the long-range spatial dependencies in the synthetic images. Results on unpaired brain and abdomen MR-to-CT image synthesis show that our method produces better synthetic CT images in both accuracy and visual quality as compared to other unsupervised synthesis methods. We also show that an approximate affine pre-registration for unpaired training data can improve synthesis results.

Radiomics in Breast Imaging from Techniques to Clinical Applications: A Review

Recent advances in computer technology have generated a new area of research known as radiomics. Radiomics is defined as the high throughput extraction and analysis of quantitative features from imaging data. Radiomic features provide information on the gray-scale patterns, inter-pixel relationships, as well as shape and spectral properties of radiological images. Moreover, these features can be used to develop computational models that may serve as a tool for personalized diagnosis and treatment guidance. Although radiomics is becoming popular and widely used in oncology, many problems such as overfitting and reproducibility issues remain unresolved. In this review, we will outline the steps of radiomics used for oncology, specifically addressing applications for breast cancer patients and focusing on technical issues.

Memory-efficient GAN-based domain translation of high resolution 3D medical images

Generative adversarial networks (GANs) are currently rarely applied on 3D medical images of large size, due to their immense computational demand. The present work proposes a multi-scale patch-based GAN approach for establishing unpaired domain translation by generating 3D medical image volumes of high resolution in a memory-efficient way. The key idea to enable memory-efficient image generation is to first generate a low-resolution version of the image followed by the generation of patches of constant sizes but successively growing resolutions. To avoid patch artifacts and incorporate global information, the patch generation is conditioned on patches from previous resolution scales. Those multi-scale GANs are trained to generate realistically looking images from image sketches in order to perform an unpaired domain translation. This allows to preserve the topology of the test data and generate the appearance of the training domain data. The evaluation of the domain translation scenarios is performed on brain MRIs of size 155 × 240 × 240 and thorax CTs of size up to 512³. Compared to common patch-based approaches, the multi-resolution scheme enables better image quality and prevents patch artifacts. Also, it ensures constant GPU memory demand independent from the image size, allowing for the generation of arbitrarily large images.

Adversarial Confidence Learning for Medical Image Segmentation and Synthesis

Generative adversarial networks (GAN) are widely used in medical image analysis tasks, such as medical image segmentation and synthesis. In these works, adversarial learning is directly applied to the original supervised segmentation (synthesis) networks. The usage of adversarial learning is effective in improving visual perception performance since adversarial learning works as realistic regularization for supervised generators. However, the quantitative performance often cannot improve as much as the qualitative performance, and it can even become worse in some cases. In this paper, we explore how we can take better advantage of adversarial learning in supervised segmentation (synthesis) models and propose an adversarial confidence learning framework to better model these problems. We analyze the roles of discriminator in the classic GANs and compare them with those in supervised adversarial systems. Based on this analysis, we propose adversarial confidence learning, i.e., besides the adversarial learning for emphasizing visual perception, we use the confidence information provided by the adversarial network to enhance the design of supervised segmentation (synthesis) network. In particular, we propose using a fully convolutional adversarial network for confidence learning to provide voxel-wise and region-wise confidence information for the segmentation (synthesis) network. With these settings, we propose a difficulty-aware attention mechanism to properly handle hard samples or regions by taking structural information into consideration so that we can better deal with the irregular distribution of medical data. Furthermore, we investigate the loss functions of various GANs and propose using the binary cross entropy loss to train the proposed adversarial system so that we can retain the unlimited modeling capacity of the discriminator. Experimental results on clinical and challenge datasets show that our proposed network can achieve state-of-the-art segmentation (synthesis) accuracy. Further analysis also indicates that adversarial confidence learning can both improve the visual perception performance and the quantitative performance.

Graded Image Generation Using Stratified CycleGAN

In medical imaging, CycleGAN has been used for various image generation tasks, including image synthesis, image denoising, and data augmentation. However, when pushing the technical limits of medical imaging, there can be a substantial variation in image quality. Here, we demonstrate that images generated by CycleGAN can be improved through explicit grading of image quality, which we call stratified CycleGAN. In this image generation task, CycleGAN is used to upgrade the image quality and content of near-infrared fluorescent (NIRF) retinal images. After manual assignment of grading scores to a small subset of the data, semi-supervised learning is applied to propagate grades across the remainder of the data and set up the training data. These scores are embedded into the CycleGAN by adding the grading score as a conditional input to the generator and by integrating an image quality classifier into the discriminator. We validate the efficacy of the proposed stratified CycleGAN by considering pairs of NIRF images at the same retinal regions (imaged with and without correction of optical aberrations achieved using adaptive optics), with the goal being to restore image quality in aberrated images such that cellular-level detail can be obtained. Overall, stratified CycleGAN generated higher quality synthetic images than traditional CycleGAN. Evaluation of cell detection accuracy confirmed that synthetic images were faithful to ground truth images of the same cells. Across this challenging dataset, F1-score improved from 76.9 ± 5.7% when using traditional CycleGAN to 85.0±3.4% when using stratified CycleGAN. These findings demonstrate the potential of stratified Cycle-GAN to improve the synthesis of medical images that exhibit a graded variation in image quality.

Binary Classification of Alzheimer's Disease Using sMRI Imaging Modality and Deep Learning

Alzheimer's disease (AD) is an irreversible devastative neurodegenerative disorder associated with progressive impairment of memory and cognitive functions. Its early diagnosis is crucial for the development of possible future treatment option(s). Structural magnetic resonance images (sMRI) play an important role to help in understanding the anatomical changes related to AD especially in its early stages. Conventional methods require the expertise of domain experts and extract hand-picked features such as gray matter substructures and train a classifier to distinguish AD subjects from healthy subjects. Different from these methods, this paper proposes to construct multiple deep 2D convolutional neural networks (2D-CNNs) to learn the various features from local brain images which are combined to make the final classification for AD diagnosis. The whole brain image was passed through two transfer learning architectures; Inception version 3 and Xception, as well as a custom Convolutional Neural Network (CNN) built with the help of separable convolutional layers which can automatically learn the generic features from imaging data for classification. Our study is conducted using cross-sectional T1-weighted structural MRI brain images from Open Access Series of Imaging Studies (OASIS) database to maintain the size and contrast over different MRI scans. Experimental results show that the transfer learning approaches exceed the performance of non-transfer learning-based approaches demonstrating the effectiveness of these approaches for the binary AD classification task.

Synthesize High-Quality Multi-Contrast Magnetic Resonance Imaging From Multi-Echo Acquisition Using Multi-Task Deep Generative Model

Multi-echo saturation recovery sequence can provide redundant information to synthesize multi-contrast magnetic resonance imaging. Traditional synthesis methods, such as GE's MAGiC platform, employ a model-fitting approach to generate parameter-weighted contrasts. However, models' over-simplification, as well as imperfections in the acquisition, can lead to undesirable reconstruction artifacts, especially in T2-FLAIR contrast. To improve the image quality, in this study, a multi-task deep learning model is developed to synthesize multi-contrast neuroimaging jointly using both signal relaxation relationships and spatial information. Compared with previous deep learning-based synthesis, the correlation between different destination contrast is utilized to enhance reconstruction quality. To improve model generalizability and evaluate clinical significance, the proposed model was trained and tested on a large multi-center dataset, including healthy subjects and patients with pathology. Results from both quantitative comparison and clinical reader study demonstrate that the multi-task formulation leads to more efficient and accurate contrast synthesis than previous methods.

Overview of Machine Learning: Part 2: Deep Learning for Medical Image Analysis

Deep learning has contributed to solving complex problems in science and engineering. This article provides the fundamental background required to understand and develop deep learning models for medical imaging applications. The authors review the main deep learning architectures such as multilayer perceptron, convolutional neural networks, autoencoders, recurrent neural networks, and generative adversarial neural networks. They also discuss the strategies for training deep learning models when the available datasets are imbalanced or of limited size and conclude with a discussion of the obstacles and challenges hindering the deployment of deep learning solutions in clinical settings.

Predicting PET-derived myelin content from multisequence MRI for individual longitudinal analysis in multiple sclerosis

Multiple sclerosis (MS) is a demyelinating and inflammatory disease of the central nervous system (CNS). The demyelination process can be repaired by the generation of a new sheath of myelin around the axon, a process termed remyelination. In MS patients, the demyelination-remyelination cycles are highly dynamic. Over the years, magnetic resonance imaging (MRI) has been increasingly used in the diagnosis of MS and it is currently the most useful paraclinical tool to assess this diagnosis. However, conventional MRI pulse sequences are not specific for pathological mechanisms such as demyelination and remyelination. Recently, positron emission tomography (PET) with radiotracer [¹¹C]PIB has become a promising tool to measure in-vivo myelin content changes which is essential to push forward our understanding of mechanisms involved in the pathology of MS, and to monitor individual patients in the context of clinical trials focused on repair therapies. However, PET imaging is invasive due to the injection of a radioactive tracer. Moreover, it is an expensive imaging test and not offered in the majority of medical centers in the world. In this work, by using multisequence MRI, we thus propose a method to predict the parametric map of [¹¹C]PIB PET, from which we derived the myelin content changes in a longitudinal analysis of patients with MS. The method is based on the proposed conditional flexible self-attention GAN (CF-SAGAN) which is specifically adjusted for high-dimensional medical images and able to capture the relationships between the spatially separated lesional regions during the image synthesis process. Jointly applying the sketch-refinement process and the proposed attention regularization that focuses on the MS lesions, our approach is shown to outperform the state-of-the-art methods qualitatively and quantitatively. Specifically, our method demonstrated a superior performance for the prediction of myelin content at voxel-wise level. More important, our method for the prediction of myelin content changes in patients with MS shows similar clinical correlations to the PET-derived gold standard indicating the potential for clinical management of patients with MS.

High tissue contrast image synthesis via multistage attention-GAN: Application to segmenting brain MR scans

Magnetic resonance imaging (MRI) presents a detailed image of the internal organs via a magnetic field. Given MRI's non-invasive advantage in repeated imaging, the low-contrast MR images in the target area make segmentation of tissue a challenging problem. This study shows the potential advantages of synthetic high tissue contrast (HTC) images through image-to-image translation techniques. Mainly, we use a novel cycle generative adversarial network (Cycle-GAN), which provides a mechanism of attention to increase the contrast within the tissue. The attention block and training on HTC images are beneficial to our model to enhance tissue visibility. We use a multistage architecture to concentrate on a single tissue as a preliminary and filter out the irrelevant context in every stage in order to increase the resolution of HTC images. The multistage architecture reduces the gap between source and target domains and alleviates synthetic images' artefacts. We apply our HTC image synthesising method to two public datasets. In order to validate the effectiveness of these images we use HTC MR images in both end-to-end and two-stage segmentation structures. The experiments on three segmentation baselines on BraTS'18 demonstrate that joining the synthetic HTC images in the multimodal segmentation framework develops the average Dice similarity scores (DSCs) of 0.8%, 0.6%, and 0.5% respectively on the whole tumour (WT), tumour core (TC), and enhancing tumour (ET) while removing one real MRI channels from the segmentation pipeline. Moreover, segmentation of infant brain tissue in T1w MR slices through our framework improves DSCs approximately 1% in cerebrospinal fluid (CSF), grey matter (GM), and white matter (WM) compared to state-of-the-art segmentation techniques. The source code of synthesising HTC images is publicly available.

High-Resolution Breast MRI Reconstruction Using a Deep Convolutional Generative Adversarial Network

BACKGROUND

A generative adversarial network could be used for high-resolution (HR) medical image synthesis with reduced scan time.

PURPOSE

To evaluate the potential of using a deep convolutional generative adversarial network (DCGAN) for generating HR_pre and HR_post images based on their corresponding low-resolution (LR) images (LR_pre and LR_post ).

STUDY TYPE

This was a retrospective analysis of a prospectively acquired cohort.

POPULATION

In all, 224 subjects were randomly divided into 200 training subjects and an independent 24 subjects testing set.

FIELD STRENGTH/SEQUENCE

Dynamic contrast-enhanced (DCE) MRI with a 1.5T scanner.

ASSESSMENT

Three breast radiologists independently ranked the image datasets, using the DCE images as the ground truth, and reviewed the image quality of both the original LR images and the generated HR images. The BI-RADS category and conspicuity of lesions were also ranked. The inter/intracorrelation coefficients (ICCs) of mean image quality scores, lesion conspicuity scores, and Breast Imaging Reporting and Data System (BI-RADS) categories were calculated between the three readers.

STATISTICAL TEST

Wilcoxon signed-rank tests evaluated differences among the multireader ranking scores.

RESULTS

The mean overall image quality scores of the generated HR_pre and HR_post were significantly higher than those of the original LR_pre and LR_post (4.77 ± 0.41 vs. 3.27 ± 0.43 and 4.72 ± 0.44 vs. 3.23 ± 0.43, P < 0.0001, respectively, in the multireader study). The mean lesion conspicuity scores of the generated HR_pre and HR_post were significantly higher than those of the original LR_pre and LR_post (4.18 ± 0.70 vs. 3.49 ± 0.58 and 4.35 ± 0.59 vs. 3.48 ± 0.61, P < 0.001, respectively, in the multireader study). The ICCs of the image quality scores, lesion conspicuity scores, and BI-RADS categories had good agreements among the three readers (all ICCs >0.75).

DATA CONCLUSION

DCGAN was capable of generating HR of the breast from fast pre- and postcontrast LR and achieved superior quantitative and qualitative performance in a multireader study.

LEVEL OF EVIDENCE

3 TECHNICAL EFFICACY STAGE: 2 J. MAGN. RESON. IMAGING 2020;52:1852-1858.

CT-TEE Image Registration for Surgical Navigation of Congenital Heart Disease Based on a Cycle Adversarial Network

Transesophageal echocardiography (TEE) has become an essential tool in interventional cardiologist's daily toolbox which allows a continuous visualization of the movement of the visceral organ without trauma and the observation of the heartbeat in real time, due to the sensor's location at the esophagus directly behind the heart and it becomes useful for navigation during the surgery. However, TEE images provide very limited data on clear anatomically cardiac structures. Instead, computed tomography (CT) images can provide anatomical information of cardiac structures, which can be used as guidance to interpret TEE images. In this paper, we will focus on how to transfer the anatomical information from CT images to TEE images via registration, which is quite challenging but significant to physicians and clinicians due to the extreme morphological deformation and different appearance between CT and TEE images of the same person. In this paper, we proposed a learning-based method to register cardiac CT images to TEE images. In the proposed method, to reduce the deformation between two images, we introduce the Cycle Generative Adversarial Network (CycleGAN) into our method simulating TEE-like images from CT images to reduce their appearance gap. Then, we perform nongrid registration to align TEE-like images with TEE images. The experimental results on both children' and adults' CT and TEE images show that our proposed method outperforms other compared methods. It is quite noted that reducing the appearance gap between CT and TEE images can benefit physicians and clinicians to get the anatomical information of ROIs in TEE images during the cardiac surgical operation.

3-To-1 Pipeline: Restructuring Transfer Learning Pipelines for Medical Imaging Classification via Optimized GAN Synthetic Images

The difficulty of applying deep learning algorithms to biomedical imaging systems arises from a lack of training images. An existing workaround to the lack of medical training images involves pre-training deep learning models on ImageNet, a non-medical dataset with millions of training images. However, the modality of ImageNet's dataset samples consisting of natural images in RGB frequently differs from the modality of medical images, consisting largely of images in grayscale such as X-ray and MRI scan imaging. While this method may be effectively applied to non-medical tasks such as human face detection, it proves ineffective in many areas of medical imaging. Recently proposed generative models such as Generative Adversarial Networks (GANs) are able to synthesize new medical images. By utilizing generated images, we may overcome the modality gap arising from current transfer learning methods. In this paper, we propose a training pipeline which outperforms both conventional GAN-synthetic methods and transfer learning methods.

Sample-Adaptive GANs: Linking Global and Local Mappings for Cross-Modality MR Image Synthesis

Generative adversarial network (GAN) has been widely explored for cross-modality medical image synthesis. The existing GAN models usually adversarially learn a global sample space mapping from the source-modality to the target-modality and then indiscriminately apply this mapping to all samples in the whole space for prediction. However, due to the scarcity of training samples in contrast to the complicated nature of medical image synthesis, learning a single global sample space mapping that is "optimal" to all samples is very challenging, if not intractable. To address this issue, this paper proposes sample-adaptive GAN models, which not only cater for the global sample space mapping between the source- and the target-modalities but also explore the local space around each given sample to extract its unique characteristic. Specifically, the proposed sample-adaptive GANs decompose the entire learning model into two cooperative paths. The baseline path learns a common GAN model by fitting all the training samples as usual for the global sample space mapping. The new sample-adaptive path additionally models each sample by learning its relationship with its neighboring training samples and using the target-modality features of these training samples as auxiliary information for synthesis. Enhanced by this sample-adaptive path, the proposed sample-adaptive GANs are able to flexibly adjust themselves to different samples, and therefore optimize the synthesis performance. Our models have been verified on three cross-modality MR image synthesis tasks from two public datasets, and they significantly outperform the state-of-the-art methods in comparison. Moreover, the experiment also indicates that our sample-adaptive strategy could be utilized to improve various backbone GAN models. It complements the existing GANs models and can be readily integrated when needed.

Fast chemical exchange saturation transfer imaging based on PROPELLER acquisition and deep neural network reconstruction

PURPOSE

To develop a method for fast chemical exchange saturation transfer (CEST) imaging.

METHODS

The periodically rotated overlapping parallel lines enhanced reconstruction (PROPELLER) sampling scheme was introduced to shorten the acquisition time. Deep neural network was employed to reconstruct CEST contrast images. Numerical simulation and experiments on a creatine phantom, hen egg, and in vivo tumor rat brain were performed to test the feasibility of this method.

RESULTS

The results from numerical simulation and experiments show that there is no significant difference between reference images and CEST-PROPELLER reconstructed images under an acceleration factor of 8.

CONCLUSION

Although the deep neural network is trained entirely on synthesized data, it works well on reconstructing experimental data. The proof of concept study demonstrates that the combination of the PROPELLER sampling scheme and the deep neural network enables considerable acceleration of saturated image acquisition and may find applications in CEST MRI.

Abdominal synthetic CT generation from MR Dixon images using a U-net trained with 'semi-synthetic' CT data

Magnetic resonance imaging (MRI) is gaining popularity in guiding radiation treatment for intrahepatic cancers due to its superior soft tissue contrast and potential of monitoring individual motion and liver function. This study investigates a deep learning-based method that generates synthetic CT volumes from T1-weighted MR Dixon images in support of MRI-based intrahepatic radiotherapy treatment planning. Training deep neutral networks for this purpose has been challenged by mismatches between CT and MR images due to motion and different organ filling status. This work proposes to resolve such challenge by generating 'semi-synthetic' CT images from rigidly aligned CT and MR image pairs. Contrasts within skeletal elements of the 'semi-synthetic' CT images were determined from CT images, while contrasts of soft tissue and air volumes were determined from voxel-wise intensity classification results on MR images. The resulting 'semi-synthetic' CT images were paired with their corresponding MR images and used to train a simple U-net model without adversarial components. MR and CT scans of 46 patients were investigated and the proposed method was evaluated for 31 patients with clinical radiotherapy plans, using 3-fold cross validation. The averaged mean absolute errors between synthetic CT and CT images across patients were 24.10 HU for liver, 28.62 HU for spleen, 47.05 HU for kidneys, 29.79 HU for spinal cord, 105.68 HU for lungs and 110.09 HU for vertebral bodies. VMAT and IMRT plans were optimized using CT-derived electron densities, and doses were recalculated using corresponding synthetic CT-derived density grids. Resulting dose differences to planning target volumes and various organs at risk were small, with the average difference less than 0.15 Gy for all dose metrics evaluated. The similarities in both image intensity and radiation dose distributions between CT and synthetic CT volumes demonstrate the accuracy of the method and its potential in supporting MRI-only radiotherapy treatment planning.

Subsampled brain MRI reconstruction by generative adversarial neural networks

A main challenge in magnetic resonance imaging (MRI) is speeding up scan time. Beyond improving patient experience and reducing operational costs, faster scans are essential for time-sensitive imaging, such as fetal, cardiac, or functional MRI, where temporal resolution is important and target movement is unavoidable, yet must be reduced. Current MRI acquisition methods speed up scan time at the expense of lower spatial resolution and costlier hardware. We introduce a practical, software-only framework, based on deep learning, for accelerating MRI acquisition, while maintaining anatomically meaningful imaging. This is accomplished by MRI subsampling followed by estimating the missing k-space samples via generative adversarial neural networks. A generator-discriminator interplay enables the introduction of an adversarial cost in addition to fidelity and image-quality losses used for optimizing the reconstruction. Promising reconstruction results are obtained from feasible sampling patterns of up to a fivefold acceleration of diverse brain MRIs, from a large publicly available dataset of healthy adult scans as well as multimodal acquisitions of multiple sclerosis patients and dynamic contrast-enhanced MRI (DCE-MRI) sequences of stroke and tumor patients. Clinical usability of the reconstructed MRI scans is assessed by performing either lesion or healthy tissue segmentation and comparing the results to those obtained by using the original, fully sampled images. Reconstruction quality and usability of the DCE-MRI sequences is demonstrated by calculating the pharmacokinetic (PK) parameters. The proposed MRI reconstruction approach is shown to outperform state-of-the-art methods for all datasets tested in terms of the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM), as well as either the mean squared error (MSE) with respect to the PK parameters, calculated for the fully sampled DCE-MRI sequences, or the segmentation compatibility, measured in terms of Dice scores and Hausdorff distance. The code is available on GitHub.

Adversarial convolutional network for esophageal tissue segmentation on OCT images

Automatic segmentation is important for esophageal OCT image processing, which is able to provide tissue characteristics such as shape and thickness for disease diagnosis. Existing automatical segmentation methods based on deep convolutional networks may not generate accurate segmentation results due to limited training set and various layer shapes. This study proposed a novel adversarial convolutional network (ACN) to segment esophageal OCT images using a convolutional network trained by adversarial learning. The proposed framework includes a generator and a discriminator, both with U-Net alike fully convolutional architecture. The discriminator is a hybrid network that discriminates whether the generated results are real and implements pixel classification at the same time. Leveraging on the adversarial training, the discriminator becomes more powerful. In addition, the adversarial loss is able to encode high order relationships of pixels, thus eliminating the requirements of post-processing. Experiments on segmenting esophageal OCT images from guinea pigs confirmed that the ACN outperforms several deep learning frameworks in pixel classification accuracy and improves the segmentation result. The potential clinical application of ACN for detecting eosinophilic esophagitis (EoE), an esophageal disease, is also presented in the experiment.

Adversarial convolutional network for esophageal tissue segmentation on OCT images

Automatic segmentation is important for esophageal OCT image processing, which is able to provide tissue characteristics such as shape and thickness for disease diagnosis. Existing automatical segmentation methods based on deep convolutional networks may not generate accurate segmentation results due to limited training set and various layer shapes. This study proposed a novel adversarial convolutional network (ACN) to segment esophageal OCT images using a convolutional network trained by adversarial learning. The proposed framework includes a generator and a discriminator, both with U-Net alike fully convolutional architecture. The discriminator is a hybrid network that discriminates whether the generated results are real and implements pixel classification at the same time. Leveraging on the adversarial training, the discriminator becomes more powerful. In addition, the adversarial loss is able to encode high order relationships of pixels, thus eliminating the requirements of post-processing. Experiments on segmenting esophageal OCT images from guinea pigs confirmed that the ACN outperforms several deep learning frameworks in pixel classification accuracy and improves the segmentation result. The potential clinical application of ACN for detecting eosinophilic esophagitis (EoE), an esophageal disease, is also presented in the experiment.