Medical Image Synthesis with Deep Convolutional Adversarial Networks

QACL: Quartet attention aware closed-loop learning for abdominal MR-to-CT synthesis via simultaneous registration

Synthesis of computed tomography (CT) images from magnetic resonance (MR) images is an important task to overcome the lack of electron density information in MR-only radiotherapy treatment planning (RTP). Some innovative methods have been proposed for abdominal MR-to-CT synthesis. However, it is still challenging due to the large misalignment between preprocessed abdominal MR and CT images and the insufficient feature information learned by models. Although several studies have used the MR-to-CT synthesis to alleviate the difficulty of multi-modal registration, this misalignment remains unsolved when training the MR-to-CT synthesis model. In this paper, we propose an end-to-end quartet attention aware closed-loop learning (QACL) framework for MR-to-CT synthesis via simultaneous registration. Specifically, the proposed quartet attention generator and mono-modal registration network form a closed-loop to improve the performance of MR-to-CT synthesis via simultaneous registration. In particular, a quartet-attention mechanism is developed to enlarge the receptive fields in networks to extract the long-range and cross-dimension spatial dependencies. Experimental results on two independent abdominal datasets demonstrate that our QACL achieves impressive results with MAE of 55.30±10.59 HU, PSNR of 22.85±1.43 dB, and SSIM of 0.83±0.04 for synthesis, and with Dice of 0.799±0.129 for registration. The proposed QACL outperforms the state-of-the-art MR-to-CT synthesis and multi-modal registration methods.

CT synthesis from MR in the pelvic area using Residual Transformer Conditional GAN

Magnetic resonance (MR) image-guided radiation therapy is a hot topic in current radiation therapy research, which relies on MR to generate synthetic computed tomography (SCT) images for radiation therapy. Convolution-based generative adversarial networks (GAN) have achieved promising results in synthesizing CT from MR since the introduction of deep learning techniques. However, due to the local limitations of pure convolutional neural networks (CNN) structure and the local mismatch between paired MR and CT images, particularly in pelvic soft tissue, the performance of GAN in synthesizing CT from MR requires further improvement. In this paper, we propose a new GAN called Residual Transformer Conditional GAN (RTCGAN), which exploits the advantages of CNN in local texture details and Transformer in global correlation to extract multi-level features from MR and CT images. Furthermore, the feature reconstruction loss is used to further constrain the image potential features, reducing over-smoothing and local distortion of the SCT. The experiments show that RTCGAN is visually closer to the reference CT (RCT) image and achieves desirable results on local mismatch tissues. In the quantitative evaluation, the MAE, SSIM, and PSNR of RTCGAN are 45.05 HU, 0.9105, and 28.31 dB, respectively. All of them outperform other comparison methods, such as deep convolutional neural networks (DCNN), Pix2Pix, Attention-UNet, WPD-DAGAN, and HDL.

Enhancement based convolutional dictionary network with adaptive window for low-dose CT denoising

BACKGROUND

Recently, one promising approach to suppress noise/artifacts in low-dose CT (LDCT) images is the CNN-based approach, which learns the mapping function from LDCT to normal-dose CT (NDCT). However, most CNN-based methods are purely data-driven, thus lacking sufficient interpretability and often losing details.

OBJECTIVE

To solve this problem, we propose a deep convolutional dictionary learning method for LDCT denoising, in which a novel convolutional dictionary learning model with adaptive window (CDL-AW) is designed, and a corresponding enhancement-based convolutional dictionary learning network (called ECDAW-Net) is constructed to unfold the CDL-AW model iteratively using the proximal gradient descent technique.

METHODS

In detail, the adaptive window-constrained convolutional dictionary atom is proposed to alleviate spectrum leakage caused by data truncation during convolution. Furthermore, in the ECDAW-Net, a multi-scale edge extraction module that consists of LoG and Sobel convolution layers is proposed in the unfolding iteration, to supplement lost textures and details. Additionally, to further improve the detail retention ability, the ECDAW-Net is trained by the compound loss function of the pixel-level MSE loss and the proposed patch-level loss, which can assist to retain richer structural information.

RESULTS

Applying ECDAW-Net to the Mayo dataset, we obtained the highest peak signal-to-noise ratio (33.94) and sub-optimal structural similarity (0.92).

CONCLUSIONS

Compared with some state-of-art methods, the interpretable ECDAW-Net performs well in suppressing noise/artifacts and preserving textures of tissue.

Towards 2D/3D Registration of the Preoperative MRI to Intraoperative Fluoroscopic Images for Visualization of Bone Defects

Magnetic Resonance Imaging (MRI) is a medical imaging modality that allows for the evaluation of soft-tissue diseases and the assessment of bone quality. Preoperative MRI volumes are used by surgeons to identify defected bones, perform the segmentation of lesions, and generate surgical plans before the surgery. Nevertheless, conventional intraoperative imaging modalities such as fluoroscopy are less sensitive in detecting potential lesions. In this work, we propose a 2D/3D registration pipeline that aims to register preoperative MRI with intraoperative 2D fluoroscopic images. To showcase the feasibility of our approach, we use the core decompression procedure as a surgical example to perform 2D/3D femur registration. The proposed registration pipeline is evaluated using digitally reconstructed radiographs (DRRs) to simulate the intraoperative fluoroscopic images. The resulting transformation from the registration is later used to create overlays of preoperative MRI annotations and planning data to provide intraoperative visual guidance to surgeons. Our results suggest that the proposed registration pipeline is capable of achieving reasonable transformation between MRI and digitally reconstructed fluoroscopic images for intraoperative visualization applications.

Semi-Supervised Learning of MRI Synthesis Without Fully-Sampled Ground Truths

Learning-based translation between MRI contrasts involves supervised deep models trained using high-quality source- and target-contrast images derived from fully-sampled acquisitions, which might be difficult to collect under limitations on scan costs or time. To facilitate curation of training sets, here we introduce the first semi-supervised model for MRI contrast translation (ssGAN) that can be trained directly using undersampled k-space data. To enable semi-supervised learning on undersampled data, ssGAN introduces novel multi-coil losses in image, k-space, and adversarial domains. The multi-coil losses are selectively enforced on acquired k-space samples unlike traditional losses in single-coil synthesis models. Comprehensive experiments on retrospectively undersampled multi-contrast brain MRI datasets are provided. Our results demonstrate that ssGAN yields on par performance to a supervised model, while outperforming single-coil models trained on coil-combined magnitude images. It also outperforms cascaded reconstruction-synthesis models where a supervised synthesis model is trained following self-supervised reconstruction of undersampled data. Thus, ssGAN holds great promise to improve the feasibility of learning-based multi-contrast MRI synthesis.

Using Simulated Training Data of Voxel-Level Generative Models to Improve 3D Neuron Reconstruction

Reconstructing neuron morphologies from fluorescence microscope images plays a critical role in neuroscience studies. It relies on image segmentation to produce initial masks either for further processing or final results to represent neuronal morphologies. This has been a challenging step due to the variation and complexity of noisy intensity patterns in neuron images acquired from microscopes. Whereas progresses in deep learning have brought the goal of accurate segmentation much closer to reality, creating training data for producing powerful neural networks is often laborious. To overcome the difficulty of obtaining a vast number of annotated data, we propose a novel strategy of using two-stage generative models to simulate training data with voxel-level labels. Trained upon unlabeled data by optimizing a novel objective function of preserving predefined labels, the models are able to synthesize realistic 3D images with underlying voxel labels. We showed that these synthetic images could train segmentation networks to obtain even better performance than manually labeled data. To demonstrate an immediate impact of our work, we further showed that segmentation results produced by networks trained upon synthetic data could be used to improve existing neuron reconstruction methods.

A Computerized Analysis with Machine Learning Techniques for the Diagnosis of Parkinson's Disease: Past Studies and Future Perspectives

According to the World Health Organization (WHO), Parkinson's disease (PD) is a neurodegenerative disease of the brain that causes motor symptoms including slower movement, rigidity, tremor, and imbalance in addition to other problems like Alzheimer's disease (AD), psychiatric problems, insomnia, anxiety, and sensory abnormalities. Techniques including artificial intelligence (AI), machine learning (ML), and deep learning (DL) have been established for the classification of PD and normal controls (NC) with similar therapeutic appearances in order to address these problems and improve the diagnostic procedure for PD. In this article, we examine a literature survey of research articles published up to September 2022 in order to present an in-depth analysis of the use of datasets, various modalities, experimental setups, and architectures that have been applied in the diagnosis of subjective disease. This analysis includes a total of 217 research publications with a list of the various datasets, methodologies, and features. These findings suggest that ML/DL methods and novel biomarkers hold promising results for application in medical decision-making, leading to a more methodical and thorough detection of PD. Finally, we highlight the challenges and provide appropriate recommendations on selecting approaches that might be used for subgrouping and connection analysis with structural magnetic resonance imaging (sMRI), DaTSCAN, and single-photon emission computerized tomography (SPECT) data for future Parkinson's research.

TCGAN: a transformer-enhanced GAN for PET synthetic CT

Multimodal medical images can be used in a multifaceted approach to resolve a wide range of medical diagnostic problems. However, these images are generally difficult to obtain due to various limitations, such as cost of capture and patient safety. Medical image synthesis is used in various tasks to obtain better results. Recently, various studies have attempted to use generative adversarial networks for missing modality image synthesis, making good progress. In this study, we propose a generator based on a combination of transformer network and a convolutional neural network (CNN). The proposed method can combine the advantages of transformers and CNNs to promote a better detail effect. The network is designed for positron emission tomography (PET) to computer tomography synthesis, which can be used for PET attenuation correction. We also experimented on two datasets for magnetic resonance T1- to T2-weighted image synthesis. Based on qualitative and quantitative analyses, our proposed method outperforms the existing methods.

Deep-learning synthesized pseudo-CT for MR high-resolution pediatric cranial bone imaging (MR-HiPCB)

PURPOSE

CT is routinely used to detect cranial abnormalities in pediatric patients with head trauma or craniosynostosis. This study aimed to develop a deep learning method to synthesize pseudo-CT (pCT) images for MR high-resolution pediatric cranial bone imaging to eliminating ionizing radiation from CT.

METHODS

3D golden-angle stack-of-stars MRI were obtained from 44 pediatric participants. Two patch-based residual UNets were trained using paired MR and CT patches randomly selected from the whole head (NetWH) or in the vicinity of bone, fractures/sutures, or air (NetBA) to synthesize pCT. A third residual UNet was trained to generate a binary brain mask using only MRI. The pCT images from NetWH (pCTNetWH ) in the brain area and NetBA (pCTNetBA ) in the nonbrain area were combined to generate pCTCom . A manual processing method using inverted MR images was also employed for comparison.

RESULTS

pCTCom (68.01 ± 14.83 HU) had significantly smaller mean absolute errors (MAEs) than pCTNetWH (82.58 ± 16.98 HU, P < 0.0001) and pCTNetBA (91.32 ± 17.2 HU, P < 0.0001) in the whole head. Within cranial bone, the MAE of pCTCom (227.92 ± 46.88 HU) was significantly lower than pCTNetWH (287.85 ± 59.46 HU, P < 0.0001) but similar to pCTNetBA (230.20 ± 46.17 HU). Dice similarity coefficient of the segmented bone was significantly higher in pCTCom (0.90 ± 0.02) than in pCTNetWH (0.86 ± 0.04, P < 0.0001), pCTNetBA (0.88 ± 0.03, P < 0.0001), and inverted MR (0.71 ± 0.09, P < 0.0001). Dice similarity coefficient from pCTCom demonstrated significantly reduced age dependence than inverted MRI. Furthermore, pCTCom provided excellent suture and fracture visibility comparable to CT.

CONCLUSION

MR high-resolution pediatric cranial bone imaging may facilitate the clinical translation of a radiation-free MR cranial bone imaging method for pediatric patients.

Synthetic augmentation for semantic segmentation of class imbalanced biomedical images: A data pair generative adversarial network approach

In recent years, deep learning (DL) has been recognized very useful in the semantic segmentation of biomedical images. Such an application, however, is significantly hindered by the lack of pixel-wise annotations. In this work, we propose a data pair generative adversarial network (DPGAN) for the purpose of synthesizing concurrently the diverse biomedical images and the segmentation labels from random latent vectors. First, a hierarchical structure is constructed consisting of three variational auto-encoder generative adversarial networks (VAEGANs) with an extra discriminator. Subsequently, to alleviate the influence from the imbalance between lesions and non-lesions areas in biomedical segmentation data sets, we divide the DPGAN into three stages, namely, background stage, mask stage and advanced stage, with each stage deploying a VAEGAN. In such a way, a large number of new segmentation data pairs are generated from random latent vectors and then used to augment the original data sets. Finally, to validate the effectiveness of the proposed DPGAN, experiments are carried out on a vestibular schwannoma data set, a kidney tumor data set and a skin cancer data set. The results indicate that, in comparison to other state-of-the-art GAN-based methods, the proposed DPGAN shows better performance in the generative quality, and meanwhile, gains an effective boost on semantic segmentation of class imbalanced biomedical images.

Medical image translation using an edge-guided generative adversarial network with global-to-local feature fusion

In this paper, we propose a framework based deep learning for medical image translation using paired and unpaired training data. Initially, a deep neural network with an encoder-decoder structure is proposed for image-to-image translation using paired training data. A multi-scale context aggregation approach is then used to extract various features from different levels of encoding, which are used during the corresponding network decoding stage. At this point, we further propose an edge-guided generative adversarial network for image-to-image translation based on unpaired training data. An edge constraint loss function is used to improve network performance in tissue boundaries. To analyze framework performance, we conducted five different medical image translation tasks. The assessment demonstrates that the proposed deep learning framework brings significant improvement beyond state-of-the-arts.

Application of cascaded GAN based on CT scan in the diagnosis of aortic dissection

PURPOSE

Currently, Computed Tomography Angiography (CTA) is the most commonly used clinical method for the diagnosis of aortic dissection, which is much better than plain CT. However, CTA examination has some disadvantages such as time-consuming image processing, complicated procedure and injection of developer. CT plain scanning is widely used in the early diagnosis of arterial dissection because of its convenience, speed and popularity. In order not to delay the optimal diagnosis and treatment time of patients, we use deep learning technology and network model to synthesize plain CT images into CTA images. Patients can be timely professional related departments of clinical diagnosis and treatment, and reduce the rate of missed diagnosis. In this paper, we propose a CTA image synthesis technique for cardiac aortic dissection based on the cascaded generative adjunctive network model.

METHOD

Firstly, we registered CT images, and then used nnU-Net segmentation network model to obtain CT and CTA paired images containing only the aorta. Then we proposed a CTA image synthesis method for aortic dissection based on cascaded generative adversarial. The core idea is to build a cascade generator and double discriminator network based on DCT channel attention mechanism to further enhance the synthesis effect of CTA.

RESULTS

The model is trained and tested on CT plain scan and CTA image data set of aortic dissection. The results show that the proposed model achieves good results in CTA image synthesis. In the CT data set, the nnU-Net model improves 8.63% and reduces 10.87mm errors in the key index DSC and HD, respectively, compared with the benchmark model U-Net. In CTA data set, nnU-Net model improves 10.27% and reduces 6.56mm error in key index DSC and HD, respectively, compared with benchmark model U-Net. In the synthesis task, the cascaded generative adm network is superior to Pix2pix and Pix2pixHD network models in both PSNR and SSIM, which proves that our proposed model has significant advantages.

CONCLUSION

This study provides new possibilities for CTA image synthesis of aortic dissection, and improves the accuracy and efficiency of diagnosis, and hopes to provide substantial help for the diagnosis of aortic dissection.

Automatic generation of retinal optical coherence tomography images based on generative adversarial networks

SIGNIFICANCE

The automatic generation algorithm of optical coherence tomography (OCT) images based on generative adversarial networks (GAN) can generate a large number of simulation images by a relatively small number of real images, which can effectively improve the classification performance.

AIM

We proposed an automatic generation algorithm for retinal OCT images based on GAN to alleviate the problem of insufficient images with high quality in deep learning, and put the diagnosis algorithm toward clinical application.

APPROACH

We designed a generation network based on GAN and trained the network with a data set constructed by 2014_BOE_Srinivasan and OCT2017 to acquire three models. Then, we generated a large number of images by the three models to augment age-related macular degeneration (AMD), diabetic macular edema (DME), and normal images. We evaluated the generated images by subjective visual observation, Fréchet inception distance (FID) scores, and a classification experiment.

RESULTS

Visual observation shows that the generated images have clear and similar features compared with the real images. Also, the lesion regions containing similar features in the real image and the generated image are randomly distributed in the image field of view. When the FID scores of the three types of generated images are lowest, three local optimal models are obtained for AMD, DME, and normal images, indicating the generated images have high quality and diversity. Moreover, the classification experiment results show that the model performance trained with the mixed images is better than that of the model trained with real images, in which the accuracy, sensitivity, and specificity are improved by 5.56%, 8.89%, and 2.22%. In addition, compared with the generation method based on variational auto-encoder (VAE), the method improved the accuracy, sensitivity, and specificity by 1.97%, 2.97%, and 0.99%, for the same test set.

CONCLUSIONS

The results show that our method can augment the three kinds of OCT images, not only effectively alleviating the problem of insufficient images with high quality but also improving the diagnosis performance.

ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis

Generative adversarial models with convolutional neural network (CNN) backbones have recently been established as state-of-the-art in numerous medical image synthesis tasks. However, CNNs are designed to perform local processing with compact filters, and this inductive bias compromises learning of contextual features. Here, we propose a novel generative adversarial approach for medical image synthesis, ResViT, that leverages the contextual sensitivity of vision transformers along with the precision of convolution operators and realism of adversarial learning. ResViT's generator employs a central bottleneck comprising novel aggregated residual transformer (ART) blocks that synergistically combine residual convolutional and transformer modules. Residual connections in ART blocks promote diversity in captured representations, while a channel compression module distills task-relevant information. A weight sharing strategy is introduced among ART blocks to mitigate computational burden. A unified implementation is introduced to avoid the need to rebuild separate synthesis models for varying source-target modality configurations. Comprehensive demonstrations are performed for synthesizing missing sequences in multi-contrast MRI, and CT images from MRI. Our results indicate superiority of ResViT against competing CNN- and transformer-based methods in terms of qualitative observations and quantitative metrics.

Prediction of lipomatous soft tissue malignancy on MRI: comparison between machine learning applied to radiomics and deep learning

Objectives

Malignancy of lipomatous soft-tissue tumours diagnosis is suspected on magnetic resonance imaging (MRI) and requires a biopsy. The aim of this study is to compare the performances of MRI radiomic machine learning (ML) analysis with deep learning (DL) to predict malignancy in patients with lipomas oratypical lipomatous tumours.

Methods

Cohort include 145 patients affected by lipomatous soft tissue tumours with histology and fat-suppressed gadolinium contrast-enhanced T1-weighted MRI pulse sequence. Images were collected between 2010 and 2019 over 78 centres with non-uniform protocols (three different magnetic field strengths (1.0, 1.5 and 3.0 T) on 16 MR systems commercialised by four vendors (General Electric, Siemens, Philips, Toshiba)).

Two approaches have been compared: (i) ML from radiomic features with and without batch correction; and (ii) DL from images. Performances were assessed using 10 cross-validation folds from a test set and next in external validation data.

Results

The best DL model was obtained using ResNet50 (resulting into an area under the curve (AUC) of 0.87 ± 0.11 (95% CI 0.65−1). For ML/radiomics, performances reached AUCs equal to 0.83 ± 0.12 (95% CI 0.59−1) and 0.99 ± 0.02 (95% CI 0.95−1) on test cohort using gradient boosting without and with batch effect correction, respectively. On the external cohort, the AUC of the gradient boosting model was equal to 0.80 and for an optimised decision threshold sensitivity and specificity were equal to 100% and 32% respectively.

Conclusions

In this context of limited observations, batch-effect corrected ML/radiomics approaches outperformed DL-based models.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41747-022-00295-9.

Virtual contrast enhancement for CT scans of abdomen and pelvis

Contrast agents are commonly used to highlight blood vessels, organs, and other structures in magnetic resonance imaging (MRI) and computed tomography (CT) scans. However, these agents may cause allergic reactions or nephrotoxicity, limiting their use in patients with kidney dysfunctions. In this paper, we propose a generative adversarial network (GAN) based framework to automatically synthesize contrast-enhanced CTs directly from the non-contrast CTs in the abdomen and pelvis region. The respiratory and peristaltic motion can affect the pixel-level mapping of contrast-enhanced learning, which makes this task more challenging than other body parts. A perceptual loss is introduced to compare high-level semantic differences of the enhancement areas between the virtual contrast-enhanced and actual contrast-enhanced CT images. Furthermore, to accurately synthesize the intensity details as well as remain texture structures of CT images, a dual-path training schema is proposed to learn the texture and structure features simultaneously. Experiment results on three contrast phases (i.e. arterial, portal, and delayed phase) show the potential to synthesize virtual contrast-enhanced CTs directly from non-contrast CTs of the abdomen and pelvis for clinical evaluation.

Application and prospect for generative adversarial networks in cross-modality reconstruction of medical images

Cross-modality reconstruction of medical images refers to predicting the image from one modality to another so as to achieve more accurate personalized medicine. Generative adversarial networks is the most commonly used deep learning technique in cross-modality reconstruction. It can generate realistic images by learning features from implicit distributions that follow the distributions of real data and then reconstruct the image of another modality rapidly. With the sharp increase in clinical demand for multi-modality medical image, this technology has been widely used in the task of cross modal reconstruction between different medical image modalities, such as magnetic resonance imaging, computed tomography and positron emission computed tomography. It can achieve accurate and efficient cross-modality image reconstruction in different parts of the body, such as the brain, heart, etc. In addition, although GAN has achieved some success in cross-modality reconstruction, its stability, generalization ability, and accuracy still need further research and improvement.

A bidirectional multilayer contrastive adaptation network with anatomical structure preservation for unpaired cross-modality medical image segmentation

Multi-modal medical image segmentation has achieved great success through supervised deep learning networks. However, because of domain shift and limited annotation information, unpaired cross-modality segmentation tasks are still challenging. The unsupervised domain adaptation (UDA) methods can alleviate the segmentation degradation of cross-modality segmentation by knowledge transfer between different domains, but current methods still suffer from the problems of model collapse, adversarial training instability, and mismatch of anatomical structures. To tackle these issues, we propose a bidirectional multilayer contrastive adaptation network (BMCAN) for unpaired cross-modality segmentation. The shared encoder is first adopted for learning modality-invariant encoding representations in image synthesis and segmentation simultaneously. Secondly, to retain the anatomical structure consistency in cross-modality image synthesis, we present a structure-constrained cross-modality image translation approach for image alignment. Thirdly, we construct a bidirectional multilayer contrastive learning approach to preserve the anatomical structures and enhance encoding representations, which utilizes two groups of domain-specific multilayer perceptron (MLP) networks to learn modality-specific features. Finally, a semantic information adversarial learning approach is designed to learn structural similarities of semantic outputs for output space alignment. Our proposed method was tested on three different cross-modality segmentation tasks: brain tissue, brain tumor, and cardiac substructure segmentation. Compared with other UDA methods, experimental results show that our proposed BMCAN achieves state-of-the-art segmentation performance on the above three tasks, and it has fewer training components and better feature representations for overcoming overfitting and domain shift problems. Our proposed method can efficiently reduce the annotation burden of radiologists in cross-modality image analysis.

A deep learning-based whole-body solution for PET/MRI attenuation correction

BACKGROUND

Deep convolutional neural networks have demonstrated robust and reliable PET attenuation correction (AC) as an alternative to conventional AC methods in integrated PET/MRI systems. However, its whole-body implementation is still challenging due to anatomical variations and the limited MRI field of view. The aim of this study is to investigate a deep learning (DL) method to generate voxel-based synthetic CT (sCT) from Dixon MRI and use it as a whole-body solution for PET AC in a PET/MRI system.

MATERIALS AND METHODS

Fifteen patients underwent PET/CT followed by PET/MRI with whole-body coverage from skull to feet. We performed MRI truncation correction and employed co-registered MRI and CT images for training and leave-one-out cross-validation. The network was pretrained with region-specific images. The accuracy of the AC maps and reconstructed PET images were assessed by performing a voxel-wise analysis and calculating the quantification error in SUV obtained using DL-based sCT (PET_sCT) and a vendor-provided atlas-based method (PET_Atlas), with the CT-based reconstruction (PET_CT) serving as the reference. In addition, region-specific analysis was performed to compare the performances of the methods in brain, lung, liver, spine, pelvic bone, and aorta.

RESULTS

Our DL-based method resulted in better estimates of AC maps with a mean absolute error of 62 HU, compared to 109 HU for the atlas-based method. We found an excellent voxel-by-voxel correlation between PET_CT and PET_sCT (R² = 0.98). The absolute percentage difference in PET quantification for the entire image was 6.1% for PET_sCT and 11.2% for PET_Atlas. The regional analysis showed that the average errors and the variability for PET_sCT were lower than PET_Atlas in all regions. The largest errors were observed in the lung, while the smallest biases were observed in the brain and liver.

CONCLUSIONS

Experimental results demonstrated that a DL approach for whole-body PET AC in PET/MRI is feasible and allows for more accurate results compared with conventional methods. Further evaluation using a larger training cohort is required for more accurate and robust performance.

Multimodal image translation via deep learning inference model trained in video domain

Background

Current medical image translation is implemented in the image domain. Considering the medical image acquisition is essentially a temporally continuous process, we attempt to develop a novel image translation framework via deep learning trained in video domain for generating synthesized computed tomography (CT) images from cone-beam computed tomography (CBCT) images.

Methods

For a proof-of-concept demonstration, CBCT and CT images from 100 patients were collected to demonstrate the feasibility and reliability of the proposed framework. The CBCT and CT images were further registered as paired samples and used as the input data for the supervised model training. A vid2vid framework based on the conditional GAN network, with carefully-designed generators, discriminators and a new spatio-temporal learning objective, was applied to realize the CBCT–CT image translation in the video domain. Four evaluation metrics, including mean absolute error (MAE), peak signal-to-noise ratio (PSNR), normalized cross-correlation (NCC), and structural similarity (SSIM), were calculated on all the real and synthetic CT images from 10 new testing patients to illustrate the model performance.

Results

The average values for four evaluation metrics, including MAE, PSNR, NCC, and SSIM, are 23.27 ± 5.53, 32.67 ± 1.98, 0.99 ± 0.0059, and 0.97 ± 0.028, respectively. Most of the pixel-wise hounsfield units value differences between real and synthetic CT images are within 50. The synthetic CT images have great agreement with the real CT images and the image quality is improved with lower noise and artifacts compared with CBCT images.

Conclusions

We developed a deep-learning-based approach to perform the medical image translation problem in the video domain. Although the feasibility and reliability of the proposed framework were demonstrated by CBCT–CT image translation, it can be easily extended to other types of medical images. The current results illustrate that it is a very promising method that may pave a new path for medical image translation research.

Deep learning-based attenuation correction for whole-body PET - a multi-tracer study with 18F-FDG, 68 Ga-DOTATATE, and 18F-Fluciclovine

A novel deep learning (DL)-based attenuation correction (AC) framework was applied to clinical whole-body oncology studies using 18F-FDG, 68 Ga-DOTATATE, and 18F-Fluciclovine. The framework used activity (λ-MLAA) and attenuation (µ-MLAA) maps estimated by the maximum likelihood reconstruction of activity and attenuation (MLAA) algorithm as inputs to a modified U-net neural network with a novel imaging physics-based loss function to learn a CT-derived attenuation map (µ-CT).

METHODS

Clinical whole-body PET/CT datasets of 18F-FDG (N = 113), 68 Ga-DOTATATE (N = 76), and 18F-Fluciclovine (N = 90) were used to train and test tracer-specific neural networks. For each tracer, forty subjects were used to train the neural network to predict attenuation maps (µ-DL). µ-DL and µ-MLAA were compared to the gold-standard µ-CT. PET images reconstructed using the OSEM algorithm with µ-DL (OSEMDL) and µ-MLAA (OSEMMLAA) were compared to the CT-based reconstruction (OSEMCT). Tumor regions of interest were segmented by two radiologists and tumor SUV and volume measures were reported, as well as evaluation using conventional image analysis metrics.

RESULTS

µ-DL yielded high resolution and fine detail recovery of the attenuation map, which was superior in quality as compared to µ-MLAA in all metrics for all tracers. Using OSEMCT as the gold-standard, OSEMDL provided more accurate tumor quantification than OSEMMLAA for all three tracers, e.g., error in SUVmax for OSEMMLAA vs. OSEMDL: - 3.6 ± 4.4% vs. - 1.7 ± 4.5% for 18F-FDG (N = 152), - 4.3 ± 5.1% vs. 0.4 ± 2.8% for 68 Ga-DOTATATE (N = 70), and - 7.3 ± 2.9% vs. - 2.8 ± 2.3% for 18F-Fluciclovine (N = 44). OSEMDL also yielded more accurate tumor volume measures than OSEMMLAA, i.e., - 8.4 ± 14.5% (OSEMMLAA) vs. - 3.0 ± 15.0% for 18F-FDG, - 14.1 ± 19.7% vs. 1.8 ± 11.6% for 68 Ga-DOTATATE, and - 15.9 ± 9.1% vs. - 6.4 ± 6.4% for 18F-Fluciclovine.

CONCLUSIONS

The proposed framework provides accurate and robust attenuation correction for whole-body 18F-FDG, 68 Ga-DOTATATE and 18F-Fluciclovine in tumor SUV measures as well as tumor volume estimation. The proposed method provides clinically equivalent quality as compared to CT in attenuation correction for the three tracers.

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

BACKGROUND AND OBJECTIVE

Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species.

METHODS

We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-of-the-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation.

RESULTS

We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets.

CONCLUSIONS

The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels, monitoring our proposed morphology-based metric is an intuitive and effective way to stop the training process and select in average optimal models.

Cross-Modality Image Adaptation Based on Volumetric Intensity Gaussian Process Models (VIGPM)

Image-based diagnosis routinely depends on more that one image modality for exploiting the complementary information they provide. However, it is not always possible to obtain images from a secondary modality for several reasons such as cost, degree of invasiveness and non-availability of scanners. Three-dimensional (3D) morphable models have made a significant contribution to the field of medical imaging for feature-based analysis. Here we extend their use to encode 3D volumetric imaging modalities. Specifically, we build a Gaussian Process (GP) over transformations establishing anatomical correspondence between training images within a modality. Given, two different modalities, the GP's eigenspace (latent space) can then be used to provide a parametric representation of each image modality, and we provide an operator for cross-domain translation between the two. We show that the latent space yields samples that are representative of the encoded modality. We also demonstrate that a 3D volumetric image can be efficiently encoded in latent space and transferred to synthesize the corresponding image in another modality. The framework called VIGPM can be extended by designing a fitting process to learn an observation in a given modality and performing cross-modality synthesis. Clinical Relevance- The proposed method provides a way to access a multi modality image from one modality. Both the source and synthetic modalities are in anatomical correspondence giving access to registered complementary information.

Brain tumor image generation using an aggregation of GAN models with style transfer

In the recent past, deep learning-based models have achieved tremendous success in computer vision-related tasks with the help of large-scale annotated datasets. An interesting application of deep learning is synthetic data generation, especially in the domain of medical image analysis. The need for such a task arises due to the scarcity of original data. Class imbalance is another reason for applying data augmentation techniques. Generative Adversarial Networks (GANs) are beneficial for synthetic image generation in various fields. However, stand-alone GANs may only fetch the localized features in the latent representation of an image, whereas combining different GANs might understand the distributed features. To this end, we have proposed AGGrGAN, an aggregation of three base GAN models-two variants of Deep Convolutional Generative Adversarial Network (DCGAN) and a Wasserstein GAN (WGAN) to generate synthetic MRI scans of brain tumors. Further, we have applied the style transfer technique to enhance the image resemblance. Our proposed model efficiently overcomes the limitation of data unavailability and can understand the information variance in multiple representations of the raw images. We have conducted all the experiments on the two publicly available datasets - the brain tumor dataset and the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2020 dataset. Results show that the proposed model can generate fine-quality images with maximum Structural Similarity Index Measure (SSIM) scores of 0.57 and 0.83 on the said two datasets.

Privacy-Preserving Artificial Intelligence Techniques in Biomedicine

BACKGROUND

Artificial intelligence (AI) has been successfully applied in numerous scientific domains. In biomedicine, AI has already shown tremendous potential, e.g., in the interpretation of next-generation sequencing data and in the design of clinical decision support systems.

OBJECTIVES

However, training an AI model on sensitive data raises concerns about the privacy of individual participants. For example, summary statistics of a genome-wide association study can be used to determine the presence or absence of an individual in a given dataset. This considerable privacy risk has led to restrictions in accessing genomic and other biomedical data, which is detrimental for collaborative research and impedes scientific progress. Hence, there has been a substantial effort to develop AI methods that can learn from sensitive data while protecting individuals' privacy.

METHOD

This paper provides a structured overview of recent advances in privacy-preserving AI techniques in biomedicine. It places the most important state-of-the-art approaches within a unified taxonomy and discusses their strengths, limitations, and open problems.

CONCLUSION

As the most promising direction, we suggest combining federated machine learning as a more scalable approach with other additional privacy-preserving techniques. This would allow to merge the advantages to provide privacy guarantees in a distributed way for biomedical applications. Nonetheless, more research is necessary as hybrid approaches pose new challenges such as additional network or computation overhead.

Multi-Conditional Constraint Generative Adversarial Network-Based MR Imaging from CT Scan Data

Magnetic resonance (MR) imaging is an important computer-aided diagnosis technique with rich pathological information. The factor of physical and physiological constraint seriously affects the applicability of that technique. Thus, computed tomography (CT)-based radiotherapy is more popular on account of its imaging rapidity and environmental simplicity. Therefore, it is of great theoretical and practical significance to design a method that can construct an MR image from the corresponding CT image. In this paper, we treat MR imaging as a machine vision problem and propose a multi-conditional constraint generative adversarial network (GAN) for MR imaging from CT scan data. Considering reversibility of GAN, both generator and reverse generator are designed for MR and CT imaging, respectively, which can constrain each other and improve consistency between features of CT and MR images. In addition, we innovatively treat the real and generated MR image discrimination as object re-identification; cosine error fusing with original GAN loss is designed to enhance verisimilitude and textural features of the MR image. The experimental results with the challenging public CT-MR image dataset show distinct performance improvement over other GANs utilized in medical imaging and demonstrate the effect of our method for medical image modal transformation.

An overview of synthetic administrative data for research

Use of administrative data for research and for planning services has increased over recent decades due to the value of the large, rich information available. However, concerns about the release of sensitive or personal data and the associated disclosure risk can lead to lengthy approval processes and restricted data access. This can delay or prevent the production of timely evidence. A promising solution to facilitate more efficient data access is to create synthetic versions of the original datasets which are less likely to hold confidential information and can minimise disclosure risk. Such data may be used as an interim solution, allowing researchers to develop their analysis plans on non-disclosive data, whilst waiting for access to the real data. We aim to provide an overview of the background and uses of synthetic data and describe common methods used to generate synthetic data in the context of UK administrative research. We propose a simplified terminology for categories of synthetic data (univariate, multivariate, and complex modality synthetic data) as well as a more comprehensive description of the terminology used in the existing literature and illustrate challenges and future directions for research.

The stability of oncologic MRI radiomic features and the potential role of deep learning: a review

The use of MRI radiomic models for the diagnosis, prognosis and treatment response prediction of tumors has been increasingly reported in literature. However, its widespread adoption in clinics is hampered by issues related to features stability. In the MRI radiomic workflow, the main factors that affect radiomic features computation can be found in the image acquisition and reconstruction phase, in the image pre-processing steps, and in the segmentation of the region of interest on which radiomic indices are extracted. Deep Neural Networks (DNNs), having shown their potentiality in the medical image processing and analysis field, can be seen as an attractive strategy to partially overcome the issues related to radiomic stability and mitigate their impact. In fact, DNN approaches can be prospectively integrated in the MRI radiomic workflow to improve image quality, obtain accurate and reproducible segmentations and generate standardized images. In this review, DNN methods that can be included in the image processing steps of the radiomic workflow are described and discussed, in the light of a detailed analysis of the literature in the context of MRI radiomic reliability.

Understanding the Research Landscape of Deep Learning in Biomedical Science: Scientometric Analysis

BACKGROUND

Advances in biomedical research using deep learning techniques have generated a large volume of related literature. However, there is a lack of scientometric studies that provide a bird's-eye view of them. This absence has led to a partial and fragmented understanding of the field and its progress.

OBJECTIVE

This study aimed to gain a quantitative and qualitative understanding of the scientific domain by analyzing diverse bibliographic entities that represent the research landscape from multiple perspectives and levels of granularity.

METHODS

We searched and retrieved 978 deep learning studies in biomedicine from the PubMed database. A scientometric analysis was performed by analyzing the metadata, content of influential works, and cited references.

RESULTS

In the process, we identified the current leading fields, major research topics and techniques, knowledge diffusion, and research collaboration. There was a predominant focus on applying deep learning, especially convolutional neural networks, to radiology and medical imaging, whereas a few studies focused on protein or genome analysis. Radiology and medical imaging also appeared to be the most significant knowledge sources and an important field in knowledge diffusion, followed by computer science and electrical engineering. A coauthorship analysis revealed various collaborations among engineering-oriented and biomedicine-oriented clusters of disciplines.

CONCLUSIONS

This study investigated the landscape of deep learning research in biomedicine and confirmed its interdisciplinary nature. Although it has been successful, we believe that there is a need for diverse applications in certain areas to further boost the contributions of deep learning in addressing biomedical research problems. We expect the results of this study to help researchers and communities better align their present and future work.

Prior information-based high-resolution tomography image reconstruction from a single digitally reconstructed radiograph

Tomography images are essential for clinical diagnosis and trauma surgery, allowing doctors to understand the internal information of patients in more detail. Since the large amount of x-ray radiation from the continuous imaging during the process of computed tomography scanning can cause serious harm to the human body, reconstructing tomographic images from sparse views becomes a potential solution to this problem. Here we present a deep-learning framework for tomography image reconstruction, namely TIReconNet, which defines image reconstruction as a data-driven supervised learning task that allows a mapping between the 2D projection view and the 3D volume to emerge from corpus. The proposed framework consists of four parts: feature extraction module, shape mapping module, volume generation module and super resolution module. The proposed framework combines 2D and 3D operations, which can generate high-resolution tomographic images with a relatively small amount of computing resources and maintain spatial information. The proposed method is verified on chest digitally reconstructed radiographs, and the reconstructed tomography images have achieved PSNR value of 18.621 ± 1.228 dB and SSIM value of 0.872 ± 0.041 when compared against the ground truth. In conclusion, an innovative convolutional neural network architecture is proposed and validated in this study, which proves that there is the potential to generate a 3D high-resolution tomographic image from a single 2D image using deep learning. This method may actively promote the application of reconstruction technology for radiation reduction, and further exploration of intraoperative guidance in trauma and orthopedics.

MRA-free intracranial vessel localization on MR vessel wall images

Analysis of vessel morphology is important in assessing intracranial atherosclerosis disease (ICAD). Recently, magnetic resonance (MR) vessel wall imaging (VWI) has been introduced to image ICAD and characterize morphology for atherosclerotic lesions. In order to automatically perform quantitative analysis on VWI data, MR angiography (MRA) acquired in the same imaging session is typically used to localize the vessel segments of interest. However, MRA may be unavailable caused by the lack or failure of the sequence in a VWI protocol. This study aims to investigate the feasibility to infer the vessel location directly from VWI. We propose to synergize an atlas-based method to preserve general vessel structure topology with a deep learning network in the motion field domain to correct the residual geometric error. Performance is quantified by examining the agreement between the extracted vessel structures from the pair-acquired and alignment-corrected angiogram, and the estimated output using a cross-validation scheme. Our proposed pipeline yields clinically feasible performance in localizing intracranial vessels, demonstrating the promise of performing vessel morphology analysis using VWI alone.

BPGAN: Brain PET synthesis from MRI using generative adversarial network for multi-modal Alzheimer's disease diagnosis

BACKGROUND AND OBJECTIVE

Multi-modal medical images, such as magnetic resonance imaging (MRI) and positron emission tomography (PET), have been widely used for the diagnosis of brain disorder diseases like Alzheimer's disease (AD) since they can provide various information. PET scans can detect cellular changes in organs and tissues earlier than MRI. Unlike MRI, PET data is difficult to acquire due to cost, radiation, or other limitations. Moreover, PET data is missing for many subjects in the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. To solve this problem, a 3D end-to-end generative adversarial network (named BPGAN) is proposed to synthesize brain PET from MRI scans, which can be used as a potential data completion scheme for multi-modal medical image research.

METHODS

We propose BPGAN, which learns an end-to-end mapping function to transform the input MRI scans to their underlying PET scans. First, we design a 3D multiple convolution U-Net (MCU) generator architecture to improve the visual quality of synthetic results while preserving the diverse brain structures of different subjects. By further employing a 3D gradient profile (GP) loss and structural similarity index measure (SSIM) loss, the synthetic PET scans have higher-similarity to the ground truth. In this study, we explore alternative data partitioning ways to study their impact on the performance of the proposed method in different medical scenarios.

RESULTS

We conduct experiments on a publicly available ADNI database. The proposed BPGAN is evaluated by mean absolute error (MAE), peak-signal-to-noise-ratio (PSNR) and SSIM, superior to other compared models in these quantitative evaluation metrics. Qualitative evaluations also validate the effectiveness of our approach. Additionally, combined with MRI and our synthetic PET scans, the accuracies of multi-class AD diagnosis on dataset-A and dataset-B are 85.00% and 56.47%, which have been improved by about 1% and 1%, respectively, compared to the stand-alone MRI.

CONCLUSIONS

The experimental results of quantitative measures, qualitative displays, and classification evaluation demonstrate that the synthetic PET images by BPGAN are reasonable and high-quality, which provide complementary information to improve the performance of AD diagnosis. This work provides a valuable reference for multi-modal medical image analysis.

Short-Axis PET Image Quality Improvement by Attention CycleGAN Using Total-Body PET

The quality of positron emission tomography (PET) imaging is positively correlated with scanner sensitivity, which is closely related to the axial field of view (FOV). Conventional short-axis PET scanners (200–350 mm FOV) reduce the imaging quality during fast scanning (2–3 minutes) due to the limitation of FOV, which reduce the reliability of diagnosis. To overcome hardware limitations and improve the image quality of short-axis PET scanners, we propose a supervised deep learning model, CycleAGAN, which is based on a cycle-consistent adversarial network (CycleGAN). We introduced the attention mechanism into the generator and focus on channel and spatial representative features and supervised learning using pairs of data to maintain the spatial consistency of the generated images with the ground truth. The imaging information of 386 patients from Henan Provincial People's Hospital was prospectively included as the dataset in this study. The training data come from the total-body PET scanner uEXPLORER. The proposed CycleAGAN is compared with traditional gray-level-based methods and learning-based methods. The results confirm that CycleAGAN achieved the best results on SSIM and NRMSE and achieved the closest distribution to ground truth in expert rating. The proposed method is not only able to improve the image quality of PET scanners with 320 mm FOV but also achieved good results on shorter FOV scanners. Patients and radiologists can benefit from the computer-aided diagnosis (CAD) system integrated with CycleAGAN.

Disentangled representation and cross-modality image translation based unsupervised domain adaptation method for abdominal organ segmentation

PURPOSE

Existing medical image segmentation models tend to achieve satisfactory performance when the training and test data are drawn from the same distribution, while they often produce significant performance degradation when used for the evaluation of cross-modality data. To facilitate the deployment of deep learning models in real-world medical scenarios and to mitigate the performance degradation caused by domain shift, we propose an unsupervised cross-modality segmentation framework based on representation disentanglement and image-to-image translation.

METHODS

Our approach is based on a multimodal image translation framework, which assumes that the latent space of images can be decomposed into a content space and a style space. First, image representations are decomposed into the content and style codes by the encoders and recombined to generate cross-modality images. Second, we propose content and style reconstruction losses to preserve consistent semantic information from original images and construct content discriminators to match the content distributions between source and target domains. Synthetic images with target domain style and source domain anatomical structures are then utilized for training of the segmentation model.

RESULTS

We applied our framework to the bidirectional adaptation experiments on MRI and CT images of abdominal organs. Compared to the case without adaptation, the Dice similarity coefficient (DSC) increased by almost 30 and 25% and average symmetric surface distance (ASSD) dropped by 13.3 and 12.2, respectively.

CONCLUSION

The proposed unsupervised domain adaptation framework can effectively improve the performance of cross-modality segmentation, and minimize the negative impact of domain shift. Furthermore, the translated image retains semantic information and anatomical structure. Our method significantly outperforms several competing methods.

DeepImageTranslator: A free, user-friendly graphical interface for image translation using deep-learning and its applications in 3D CT image analysis

The advent of deep-learning has set new standards in an array of image translation applications. At present, the use of these methods often requires computer programming experience. Non-commercial programs with graphical interface usually do not allow users to fully customize their deep-learning pipeline. Therefore, our primary objective is to provide a simple graphical interface that allows researchers with no programming experience to easily create, train, and evaluate custom deep-learning models for image translation. We also aimed to test the applicability of our tool in CT image semantic segmentation and noise reduction. DeepImageTranslator was implemented using the Tkinter library, the standard Python interface to the Tk graphical user interface toolkit; backend computations were implemented using data augmentation packages such as Pillow, Numpy, OpenCV, Augmentor, Tensorflow, and Keras libraries. Convolutional neural networks (CNNs) were trained using DeepImageTranslator. The effects of data augmentation, deep-supervision, and sample size on model accuracy were also systematically assessed. The DeepImageTranslator a simple tool that allows users to customize all aspects of their deep-learning pipeline, including the CNN, training optimizer, loss function, and the types of training image augmentation scheme. We showed that DeepImageTranslator can be used to achieve state-of-the-art accuracy and generalizability in semantic segmentation and noise reduction. Highly accurate 3D segmentation models for body composition can be obtained using training sample sizes as small as 17 images. In conclusion, an open-source deep-learning tool for accurate image translation with a user-friendly graphical interface was presented and evaluated. This standalone software can be downloaded at: https://sourceforge.net/projects/deepimagetranslator/.

Toward MR-only proton therapy planning for pediatric brain tumors: synthesis of relative proton stopping power images with multiple sequence MRI and development of an online quality assurance tool

PURPOSE

To generate synthetic relative proton-stopping-power (sRPSP) images from MRI sequence(s) and develop an online quality assurance (QA) tool for sRPSP to facilitate safe integration of MR-only proton planning into clinical practice.

MATERIALS AND METHODS

Planning CT and MR images of 195 pediatric brain tumor patients were utilized (training: 150, testing: 45). Seventeen consistent-cycle Generative Adversarial Network (ccGAN) models were trained separately using paired CT-converted RPSP and MRI datasets to transform a subject's MRI into sRPSP. T1-weighted (T1W), T2-weighted (T2W), and FLAIR MRI were permutated to form 17 combinations, with or without preprocessing, for determining the optimal training sequence(s). For evaluation, sRPSP images were converted to synthetic CT (sCT) and compared to the real CT in terms of mean absolute error (MAE) in HU. For QA, sCT was deformed and compared to a reference template built from training dataset to produce a flag map, highlighting pixels that deviate by >100 HU and fall outside the mean ± standard deviation reference intensity. The gamma intensity analysis (10%/3mm) of the deformed sCT against the QA template on the intensity difference was investigated as a surrogate of sCT accuracy.

RESULTS

The sRPSP images generated from a single T1W or T2W sequence outperformed that generated from multi-MRI sequences in terms of MAE (all P<0.05). Preprocessing with N4 bias and histogram matching reduced MAE of T2W MRI-based sCT (54±21 HU vs. 42±13 HU, P = .002). The gamma intensity analysis of sCT against the QA template was highly correlated with the MAE of sCT against the real CT in the testing cohort (r = -0.89 for T1W sCT; r = -0.93 for T2W sCT).

CONCLUSION

Accurate sRPSP images can be generated from T1W/T2W MRI for proton planning. A QA tool highlights regions of inaccuracy, flagging problematic cases unsuitable for clinical use. This article is protected by copyright. All rights reserved.

Digital Rock Reconstruction with User-Defined Properties Using Conditional Generative Adversarial Networks

Uncertainty is ubiquitous with multiphase flow in subsurface rocks due to their inherent heterogeneity and lack of in-situ measurements. To complete uncertainty analysis in a multi-scale manner, it is a prerequisite to provide sufficient rock samples. Even though the advent of digital rock technology offers opportunities to reproduce rocks, it still cannot be utilized to provide massive samples due to its high cost, thus leading to the development of diversified mathematical methods. Among them, two-point statistics (TPS) and multi-point statistics (MPS) are commonly utilized, which feature incorporating low-order and high-order statistical information, respectively. Recently, generative adversarial networks (GANs) are becoming increasingly popular since they can reproduce training images with excellent visual and consequent geologic realism. However, standard GANs can only incorporate information from data, while leaving no interface for user-defined properties, and thus may limit the representativeness of reconstructed samples. In this study, we propose conditional GANs for digital rock reconstruction, aiming to reproduce samples not only similar to the real training data, but also satisfying user-specified properties. In fact, the proposed framework can realize the targets of MPS and TPS simultaneously by incorporating high-order information directly from rock images with the GANs scheme, while preserving low-order counterparts through conditioning. We conduct three reconstruction experiments, and the results demonstrate that rock type, rock porosity, and correlation length can be successfully conditioned to affect the reconstructed rock images. The randomly reconstructed samples with specified rock type, porosity and correlation length will contribute to the subsequent research on pore-scale multiphase flow and uncertainty quantification.

Brain Tumor Classification Using a Combination of Variational Autoencoders and Generative Adversarial Networks

Brain tumors are a pernicious cancer with one of the lowest five-year survival rates. Neurologists often use magnetic resonance imaging (MRI) to diagnose the type of brain tumor. Automated computer-assisted tools can help them speed up the diagnosis process and reduce the burden on the health care systems. Recent advances in deep learning for medical imaging have shown remarkable results, especially in the automatic and instant diagnosis of various cancers. However, we need a large amount of data (images) to train the deep learning models in order to obtain good results. Large public datasets are rare in medicine. This paper proposes a framework based on unsupervised deep generative neural networks to solve this limitation. We combine two generative models in the proposed framework: variational autoencoders (VAEs) and generative adversarial networks (GANs). We swap the encoder-decoder network after initially training it on the training set of available MR images. The output of this swapped network is a noise vector that has information of the image manifold, and the cascaded generative adversarial network samples the input from this informative noise vector instead of random Gaussian noise. The proposed method helps the GAN to avoid mode collapse and generate realistic-looking brain tumor magnetic resonance images. These artificially generated images could solve the limitation of small medical datasets up to a reasonable extent and help the deep learning models perform acceptably. We used the ResNet50 as a classifier, and the artificially generated brain tumor images are used to augment the real and available images during the classifier training. We compared the classification results with several existing studies and state-of-the-art machine learning models. Our proposed methodology noticeably achieved better results. By using brain tumor images generated artificially by our proposed method, the classification average accuracy improved from 72.63% to 96.25%. For the most severe class of brain tumor, glioma, we achieved 0.769, 0.837, 0.833, and 0.80 values for recall, specificity, precision, and F1-score, respectively. The proposed generative model framework could be used to generate medical images in any domain, including PET (positron emission tomography) and MRI scans of various parts of the body, and the results show that it could be a useful clinical tool for medical experts.

Multi-resolution residual deep neural network for improving pelvic CBCT image quality

PURPOSE

Cone-beam computed tomography (CBCT) is frequently used for accurate image guided radiation therapy (IGRT). However, the poor CBCT image quality prevents its further clinical use. Thus, it is important to improve the HU accuracy and structure preservation of CBCT images.

METHODS

In this study, we proposed a novel method to generate synthetic CT (sCT) images from CBCT images. A multi-resolution residual deep neural network (RDNN) was adopted for image regression from CBCT images to planning CT (pCT) images. At the coarse level, RDNN was first trained with a large amount of lower resolution images, which can make the network focus on coarse information and prevent overfitting problems. More fine information was obtained gradually by fine-tuning the coarse model using fewer number of higher resolution images. Our model was optimized by using aligned pCT and CBCT image pairs of a particular body region of 153 prostate cancer patients treated in our hospital (120 for training, 33 for testing). Five-fold cross-validation was used to tune the hyperparameters and the testing data were used to evaluate the performance of the final models.

RESULTS

The mean absolute error (MAE) between CBCT and pCT on the testing data was 352.56 HU, while the MAE between the sCT and pCT images was 52.18 HU for our proposed multi-resolution RDNN model, which reduced the MAE by 85.20% (p < 0.01). In addition, the average structural similarity index measure (SSIM) between the sCT and CBCT was 19.64% (p = 0.01) higher than that of pCT and CBCT.

CONCLUSIONS

The sCT images generated using our proposed multi-resolution RDNN have higher HU accuracy and structural fidelity, which may promote the further applications of CBCT images in the clinic for structure segmentation, dose calculation and adaptive radiotherapy planning. This article is protected by copyright. All rights reserved.