MRI super-resolution using similarity distance and multi-scale receptive field based feature fusion GAN and pre-trained slice interpolation network

Challenges arise in achieving high-resolution Magnetic Resonance Imaging (MRI) to improve disease diagnosis accuracy due to limitations in hardware, patient discomfort, long acquisition times, and high costs. While Convolutional Neural Networks (CNNs) have shown promising results in MRI super-resolution, they often don't look into the structural similarity and prior information available in consecutive MRI slices. By leveraging information from sequential slices, more robust features can be obtained, potentially leading to higher-quality MRI slices. We propose a multi-slice two-dimensional (2D) MRI super-resolution network that combines a Generative Adversarial Network (GAN) with feature fusion and a pre-trained slice interpolation network to achieve three-dimensional (3D) super-resolution. The proposed model requires consecutively acquired three low-resolution (LR) MRI slices along a specific axis, and achieves the reconstruction of the MRI slices in the remaining two axes. The network effectively enhances both in-plane and out-of-plane resolution along the sagittal axis while addressing computational and memory constraints in 3D super-resolution. The proposed generator has a in-plane and out-of-plane Attention (IOA) network that fuses both in-plane and out-plane features of MRI dynamically. In terms of out-of-plane attention, the network merges features by considering the similarity distance between features and for in-plane attention, the network employs a two-level pyramid structure with varying receptive fields to extract features at different scales, ensuring the inclusion of both global and local features. Subsequently, to achieve 3D MRI super-resolution, a pre-trained slice interpolation network is used that takes two consecutive super-resolved MRI slices to generate a new intermediate slice. To further enhance the network performance and perceptual quality, we introduce a feature up-sampling layer and a feature extraction block with Scaled Exponential Linear Unit (SeLU). Moreover, our super-resolution network incorporates VGG loss from a fine-tuned VGG-19 network to provide additional enhancement. Through experimental evaluations on the IXI dataset and BRATS dataset, using the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) and the number of training parameters, we demonstrate the superior performance of our method compared to the existing techniques. Also, the proposed model can be adapted or modified to achieve super-resolution for both 2D and 3D MRI data.

Application of intelligent tongue image analysis in Conjunction with microbiomes in the diagnosis of MAFLD

Background

Metabolic associated fatty liver disease (MAFLD) is a widespread liver disease that can lead to liver fibrosis and cirrhosis. Therefore, it is essential to develop early diagnosic and screening methods.

Methods

We performed a cross-sectional observational study. In this study, based on data from 92 patients with MAFLD and 74 healthy individuals, we observed the characteristics of tongue images, tongue coating and intestinal flora. A generative adversarial network was used to extract tongue image features, and 16S rRNA sequencing was performed using the tongue coating and intestinal flora. We then applied tongue image analysis technology combined with microbiome technology to obtain an MAFLD early screening model with higher accuracy. In addition, we compared different modelling methods, including Extreme Gradient Boosting (XGBoost), random forest, neural networks(MLP), stochastic gradient descent(SGD), and support vector machine(SVM).

Results

The results show that tongue-coating Streptococcus and Rothia, intestinal Blautia, and Streptococcus are potential biomarkers for MAFLD. The diagnostic model jointly incorporating tongue image features, basic information (gender, age, BMI), and tongue coating marker flora (Streptococcus, Rothia), can have an accuracy of 96.39%, higher than the accuracy value except for bacteria.

Conclusion

Combining computer-intelligent tongue diagnosis with microbiome technology enhances MAFLD diagnostic accuracy and provides a convenient early screening reference.

An adversarial learning approach to generate pressure support ventilation waveforms for asynchrony detection

BACKGROUND AND OBJECTIVE

Mechanical ventilation is a life-saving treatment for critically-ill patients. During treatment, patient-ventilator asynchrony (PVA) can occur, which can lead to pulmonary damage, complications, and higher mortality. While traditional detection methods for PVAs rely on visual inspection by clinicians, in recent years, machine learning models are being developed to detect PVAs automatically. However, training these models requires large labeled datasets, which are difficult to obtain, as labeling is a labour-intensive and time-consuming task, requiring clinical expertise. Simulating the lung-ventilator interactions has been proposed to obtain large labeled datasets to train machine learning classifiers. However, the obtained data lacks the influence of different hardware, of servo-controlled algorithms, and different sources of noise. Here, we propose VentGAN, an adversarial learning approach to improve simulated data by learning the ventilator fingerprints from unlabeled clinical data.

METHODS

In VentGAN, the loss functions are designed to add characteristics of clinical waveforms to the generated results, while preserving the labels of the simulated waveforms. To validate VentGAN, we compare the performance for detection and classification of PVAs when training a previously developed machine learning algorithm with the original simulated data and with the data generated by VentGAN. Testing is performed on independent clinical data labeled by experts. The McNemar test is applied to evaluate statistical differences in the obtained classification accuracy.

RESULTS

VentGAN significantly improves the classification accuracy for late cycling, early cycling and normal breaths (p< 0.01); no significant difference in accuracy was observed for delayed inspirations (p = 0.2), while the accuracy decreased for ineffective efforts (p< 0.01).

CONCLUSIONS

Generation of realistic synthetic data with labels by the proposed framework is feasible and represents a promising avenue for improving training of machine learning models.

CosSIF: Cosine similarity-based image filtering to overcome low inter-class variation in synthetic medical image datasets

Crafting effective deep learning models for medical image analysis is a complex task, particularly in cases where the medical image dataset lacks significant inter-class variation. This challenge is further aggravated when employing such datasets to generate synthetic images using generative adversarial networks (GANs), as the output of GANs heavily relies on the input data. In this research, we propose a novel filtering algorithm called Cosine Similarity-based Image Filtering (CosSIF). We leverage CosSIF to develop two distinct filtering methods: Filtering Before GAN Training (FBGT) and Filtering After GAN Training (FAGT). FBGT involves the removal of real images that exhibit similarities to images of other classes before utilizing them as the training dataset for a GAN. On the other hand, FAGT focuses on eliminating synthetic images with less discriminative features compared to real images used for training the GAN. The experimental results reveal that the utilization of either the FAGT or FBGT method reduces low inter-class variation in clinical image classification datasets and enables GANs to generate synthetic images with greater discriminative features. Moreover, modern transformer and convolutional-based models, trained with datasets that utilize these filtering methods, lead to less bias toward the majority class, more accurate predictions of samples in the minority class, and overall better generalization capabilities. Code and implementation details are available at: https://github.com/mominul-ssv/cossif.

Interactive design generation and optimization from generative adversarial networks in spatial computing

This paper focuses on exploring the application possibilities and optimization problems of Generative Adversarial Networks (GANs) in spatial computing to improve design efficiency and creativity and achieve a more intelligent design process. A method for icon generation is proposed, and a basic architecture for icon generation is constructed. A system with generation and optimization capabilities is constructed to meet various requirements in spatial design by introducing the concept of interactive design and the characteristics of requirement conditions. Next, the generated icons can effectively maintain diversity and innovation while meeting the conditional features by integrating multi-feature recognition modules into the discriminator and optimizing the structure of conditional features. The experiment uses publicly available icon datasets, including LLD-Icon and Icons-50. The icon shape generated by the model proposed here is more prominent, and the color of colored icons can be more finely controlled. The Inception Score (IS) values under different models are compared, and it is found that the IS value of the proposed model is 7.05, which is higher than that of other GAN models. The multi-feature icon generation model based on Auxiliary Classifier GANs performs well in presenting multiple feature representations of icons. After introducing multi-feature recognition modules into the network model, the peak error of the recognition network is only 2.000 in the initial stage, while the initial error of the ordinary GAN without multi-feature recognition modules is as high as 5.000. It indicates that the improved model effectively helps the discriminative network recognize the core information of icon images more quickly. The research results provide a reference basis for achieving more efficient and innovative interactive space design.

CT synthesis from MR images using frequency attention conditional generative adversarial network

Magnetic resonance (MR) image-guided radiotherapy is widely used in the treatment planning of malignant tumors, and MR-only radiotherapy, a representative of this technique, requires synthetic computed tomography (sCT) images for effective radiotherapy planning. Convolutional neural networks (CNN) have shown remarkable performance in generating sCT images. However, CNN-based models tend to synthesize more low-frequency components and the pixel-wise loss function usually used to optimize the model can result in blurred images. To address these problems, a frequency attention conditional generative adversarial network (FACGAN) is proposed in this paper. Specifically, a frequency cycle generative model (FCGM) is designed to enhance the inter-mapping between MR and CT and extract more rich tissue structure information. Additionally, a residual frequency channel attention (RFCA) module is proposed and incorporated into the generator to enhance its ability in perceiving the high-frequency image features. Finally, high-frequency loss (HFL) and cycle consistency high-frequency loss (CHFL) are added to the objective function to optimize the model training. The effectiveness of the proposed model is validated on pelvic and brain datasets and compared with state-of-the-art deep learning models. The results show that FACGAN produces higher-quality sCT images while retaining clearer and richer high-frequency texture information.

Feedback linearization control for uncertain nonlinear systems via generative adversarial networks

This article presents a novel approach to leverage generative adversarial networks(GANs) techniques to learn a feedback linearization controller(FLC) for a class of uncertain nonlinear systems. By estimating uncertainty through the adversarial process, where ground truth samples are exclusively obtained from a predefined integral model, the feedback linearization controller, learned through a minimax two-player optimization framework, enhances the reference tracking performance of the input-output uncertain nonlinear system. Furthermore, we provide theoretical guarantee of convergence and stability, demonstrating the safe recovery of robust FLC. We also address the common challenge of mode collapse in GANs training through the strict convexity of our synthesized generator structure and an enhanced adversarial loss. Comprehensive simulations and practical experiments are conducted to underscore the superiority and efficacy of our proposed approach.

ReeGAN: MRI image edge-preserving synthesis based on GANs trained with misaligned data

As a crucial medical examination technique, different modalities of magnetic resonance imaging (MRI) complement each other, offering multi-angle and multi-dimensional insights into the body's internal information. Therefore, research on MRI cross-modality conversion is of great significance, and many innovative techniques have been explored. However, most methods are trained on well-aligned data, and the impact of misaligned data has not received sufficient attention. Additionally, many methods focus on transforming the entire image and ignore crucial edge information. To address these challenges, we propose a generative adversarial network based on multi-feature fusion, which effectively preserves edge information while training on noisy data. Notably, we consider images with limited range random transformations as noisy labels and use an additional small auxiliary registration network to help the generator adapt to the noise distribution. Moreover, we inject auxiliary edge information to improve the quality of synthesized target modality images. Our goal is to find the best solution for cross-modality conversion. Comprehensive experiments and ablation studies demonstrate the effectiveness of the proposed method.

Anomaly detection using deep convolutional generative adversarial networks in the internet of things

Advanced 5 G and 6 G technologies have accelerated the adoption of the Internet of Things (IoT) and are a priority in providing support for high-speed communication and fast data analysis. One of IoT networks benefits is automated networking, which unfortunately increases the risk of security, integrity, and privacy breaches. Therefore, in this paper, we propose a weighted stacked ensemble model combining deep convolutional generative adversarial and bidirectional long short-term memory networks. The proposed model has been regularized, and hyperparameter tuning has been performed. The tuned model is then evaluated on four publicly available current IoT datasets. The proposed model exhibits significant improvement in standard performance measures for both binary and multiclass classification. Generalization error has been reduced by a rate of 0.005% and to overcome the issue of overfitting, a L2 regularization technique has been deployed. The overall Accuracy of the model on various datasets is 99.99% for BOT-IoT, 99.08% for IoT23, 99.82% for UNSWNB15, and 99.96% for ToN_IoT, respectively, alongside improvements in Precision, Recall, and F1-score.

Cardiac fat segmentation using computed tomography and an image-to-image conditional generative adversarial neural network

In recent years, research has highlighted the association between increased adipose tissue surrounding the human heart and elevated susceptibility to cardiovascular diseases such as atrial fibrillation and coronary heart disease. However, the manual segmentation of these fat deposits has not been widely implemented in clinical practice due to the substantial workload it entails for medical professionals and the associated costs. Consequently, the demand for more precise and time-efficient quantitative analysis has driven the emergence of novel computational methods for fat segmentation. This study presents a novel deep learning-based methodology that offers autonomous segmentation and quantification of two distinct types of cardiac fat deposits. The proposed approach leverages the pix2pix network, a generative conditional adversarial network primarily designed for image-to-image translation tasks. By applying this network architecture, we aim to investigate its efficacy in tackling the specific challenge of cardiac fat segmentation, despite not being originally tailored for this purpose. The two types of fat deposits of interest in this study are referred to as epicardial and mediastinal fats, which are spatially separated by the pericardium. The experimental results demonstrated an average accuracy of 99.08% and f1-score 98.73 for the segmentation of the epicardial fat and 97.90% of accuracy and f1-score of 98.40 for the mediastinal fat. These findings represent the high precision and overlap agreement achieved by the proposed methodology. In comparison to existing studies, our approach exhibited superior performance in terms of f1-score and run time, enabling the images to be segmented in real time.

EEG dataset for energy data visualizations

User behavior plays a substantial role in shaping household energy use. Nevertheless, the methodologies employed by researchers to examine user behavior exhibit certain limitations in terms of their reach. The present article introduces an openly accessible collection of electroencephalography (EEG) recordings, comprising EEG data collected from individuals who were subjected to energy data visualizations. The dataset comprises EEG recordings obtained from 28 individuals who were in good health. The EEG recordings were collected using a 32-channel EMOTIV EEG device, and the international 10-20 electrode system was employed for precise electrode placement. The energy data visualizations were generated and showcased utilizing the PsychoPy software. To ascertain the participants' affective state, they were requested to rate the valence and arousal of each stimulus through the utilization of a self-assessment manikin (SAM). Additionally, three inquiries were posed for every stimulation. The dataset includes both original data visualizations and ratings. Additionally, the raw EEG data has been divided into segments consisting of data visualizations and neutral images, with the use of event markers, in order to assist analysis. The EEG recordings were recorded and stored utilizing the EMOTIVPro application, whereas the subjective reactions were captured and preserved using the PsychoPy application. Furthermore, the generation of synthetic EEG data is accomplished by employing the Generative Adversarial Network (GAN) architecture on the acquired EEG dataset. The synthetic EEG data created is integrated with empirical EEG data, and afterwards subjected to qualitative and quantitative analysis in order to improve performance. The dataset presented herein showcases a pioneering utilization of EEG investigation and offers a valuable foundation for scholars in the domains of computer science, energy conservation, artificial intelligence, brain-computer interfaces, and human-computer interaction.

A versatile Wavelet-Enhanced CNN-Transformer for improved fluorescence microscopy image restoration

Fluorescence microscopes are indispensable tools for the life science research community. Nevertheless, the presence of optical component limitations, coupled with the maximum photon budget that the specimen can tolerate, inevitably leads to a decline in imaging quality and a lack of useful signals. Therefore, image restoration becomes essential for ensuring high-quality and accurate analyses. This paper presents the Wavelet-Enhanced Convolutional-Transformer (WECT), a novel deep learning technique developed specifically for the purpose of reducing noise in microscopy images and attaining super-resolution. Unlike traditional approaches, WECT integrates wavelet transform and inverse-transform for multi-resolution image decomposition and reconstruction, resulting in an expanded receptive field for the network without compromising information integrity. Subsequently, multiple consecutive parallel CNN-Transformer modules are utilized to collaboratively model local and global dependencies, thus facilitating the extraction of more comprehensive and diversified deep features. In addition, the incorporation of generative adversarial networks (GANs) into WECT enhances its capacity to generate high perceptual quality microscopic images. Extensive experiments have demonstrated that the WECT framework outperforms current state-of-the-art restoration methods on real fluorescence microscopy data under various imaging modalities and conditions, in terms of quantitative and qualitative analysis.

Intelligent phenotype-detection and gene expression profile generation with generative adversarial networks

Gene expression analysis is valuable for cancer type classification and identifying diverse cancer phenotypes. The latest high-throughput RNA sequencing devices have enabled access to large volumes of gene expression data. However, we face several challenges, such as data security and privacy, when we develop machine learning-based classifiers for categorizing cancer types with these datasets. To address these issues, we propose IP3G (Intelligent Phenotype-detection and Gene expression profile Generation with Generative adversarial network), a model based on Generative Adversarial Networks. IP3G tackles two major problems: augmenting gene expression data and unsupervised phenotype discovery. By converting gene expression profiles into 2-Dimensional images and leveraging IP3G, we generate new profiles for specific phenotypes. IP3G learns disentangled representations of gene expression patterns and identifies phenotypes without labeled data. We improve the objective function of the GAN used in IP3G by employing the earth mover distance and a novel mutual information function. IP3G outperforms clustering methods like k-Means, DBSCAN, and GMM in unsupervised phenotype discovery, while also surpassing SVM and CNN classification accuracy by up to 6% through gene expression profile augmentation. The source code for the developed IP3G is accessible to the public on GitHub.

A joint learning method for incomplete and imbalanced data in electronic health record based on generative adversarial networks

Electronic health records (EHR), present challenges of incomplete and imbalanced data in clinical predictions. Previous studies addressed these two issues with two-step separately, which caused the decrease in the performance of prediction tasks. In this paper, we propose a unified framework to simultaneously addresses the challenges of incomplete and imbalanced data in EHR. Based on the framework, we develop a model called Missing Value Imputation and Imbalanced Learning Generative Adversarial Network (MVIIL-GAN). We use MVIIL-GAN to perform joint learning on the imputation process of high missing rate data and the conditional generation process of EHR data. The joint learning is achieved by introducing two discriminators to distinguish the fake data from the generated data at sample-level and variable-level. MVIIL-GAN integrate the missing values imputation and data generation in one step, improving the consistency of parameter optimization and the performance of prediction tasks. We evaluate our framework using the public dataset MIMIC-IV with high missing rates data and imbalanced data. Experimental results show that MVIIL-GAN outperforms existing methods in prediction performance. The implementation of MVIIL-GAN can be found at https://github.com/Peroxidess/MVIIL-GAN.

FFPE++: Improving the quality of formalin-fixed paraffin-embedded tissue imaging via contrastive unpaired image-to-image translation

Formalin-fixation and paraffin-embedding (FFPE) is a technique for preparing and preserving tissue specimens that has been utilized in histopathology since the late 19th century. This process is further complicated by FFPE preparation steps such as fixation, processing, embedding, microtomy, staining, and coverslipping, which often results in artifacts due to the complex histological and cytological characteristics of a tissue specimen. The term "artifacts" includes, but is not limited to, staining inconsistencies, tissue folds, chattering, pen marks, blurring, air bubbles, and contamination. The presence of artifacts may interfere with pathological diagnosis in disease detection, subtyping, grading, and choice of therapy. In this study, we propose FFPE++, an unpaired image-to-image translation method based on contrastive learning with a mixed channel-spatial attention module and self-regularization loss that drastically corrects the aforementioned artifacts in FFPE tissue sections. Turing tests were performed by 10 board-certified pathologists with more than 10 years of experience. These tests which were performed for ovarian carcinoma, lung adenocarcinoma, lung squamous cell carcinoma, and papillary thyroid carcinoma, demonstrate the clear superiority of the proposed method in many clinical aspects compared with standard FFPE images. Based on the qualitative experiments and feedback from the Turing tests, we believe that FFPE++ can contribute to substantial diagnostic and prognostic accuracy in clinical pathology in the future and can also improve the performance of AI tools in digital pathology. The code and dataset are publicly available at https://github.com/DeepMIALab/FFPEPlus.

Few-shot image generation with reverse contrastive learning

Generative models, such as Generative Adversarial Networks (GANs), have recently shown remarkable capabilities in various generation tasks. However, the success of these models heavily depends on the availability of a large-scale training dataset. When the size of the training dataset is limited, the quality and diversity of the generated results suffer from severe degradation. In this paper, we propose a novel approach, Reverse Contrastive Learning (RCL), to address the problem of high-quality and diverse image generation under few-shot settings. The success of RCL benefits from a two-sided, powerful regularization. Our proposed regularization is designed based on the correlation between generated samples, which can effectively utilize the latent feature information between different levels of samples. It does not require any auxiliary information or augmentation techniques. A series of qualitative and quantitative results show that our proposed method is superior to the existing State-Of-The-Art (SOTA) methods under the few-shot setting and is still competitive under the low-shot setting, showcasing the effectiveness of RCL. Code will be released upon acceptance at https://github.com/gouayao/RCL.

SynCLay: Interactive synthesis of histology images from bespoke cellular layouts

Automated synthesis of histology images has several potential applications in computational pathology. However, no existing method can generate realistic tissue images with a bespoke cellular layout or user-defined histology parameters. In this work, we propose a novel framework called SynCLay (Synthesis from Cellular Layouts) that can construct realistic and high-quality histology images from user-defined cellular layouts along with annotated cellular boundaries. Tissue image generation based on bespoke cellular layouts through the proposed framework allows users to generate different histological patterns from arbitrary topological arrangement of different types of cells (e.g., neutrophils, lymphocytes, epithelial cells and others). SynCLay generated synthetic images can be helpful in studying the role of different types of cells present in the tumor microenvironment. Additionally, they can assist in balancing the distribution of cellular counts in tissue images for designing accurate cellular composition predictors by minimizing the effects of data imbalance. We train SynCLay in an adversarial manner and integrate a nuclear segmentation and classification model in its training to refine nuclear structures and generate nuclear masks in conjunction with synthetic images. During inference, we combine the model with another parametric model for generating colon images and associated cellular counts as annotations given the grade of differentiation and cellularities (cell densities) of different cells. We assess the generated images quantitatively using the Frechet Inception Distance and report on feedback from trained pathologists who assigned realism scores to a set of images generated by the framework. The average realism score across all pathologists for synthetic images was as high as that for the real images. Moreover, with the assistance from pathologists, we showcase the ability of the generated images to accurately differentiate between benign and malignant tumors, thus reinforcing their reliability. We demonstrate that the proposed framework can be used to add new cells to a tissue images and alter cellular positions. We also show that augmenting limited real data with the synthetic data generated by our framework can significantly boost prediction performance of the cellular composition prediction task. The implementation of the proposed SynCLay framework is available at https://github.com/Srijay/SynCLay-Framework.

Combining generative modelling and semi-supervised domain adaptation for whole heart cardiovascular magnetic resonance angiography segmentation

BACKGROUND

Quantification of three-dimensional (3D) cardiac anatomy is important for the evaluation of cardiovascular diseases. Changes in anatomy are indicative of remodeling processes as the heart tissue adapts to disease. Although robust segmentation methods exist for computed tomography angiography (CTA), few methods exist for whole-heart cardiovascular magnetic resonance angiograms (CMRA) which are more challenging due to variable contrast, lower signal to noise ratio and a limited amount of labeled data.

METHODS

Two state-of-the-art unsupervised generative deep learning domain adaptation architectures, generative adversarial networks and variational auto-encoders, were applied to 3D whole heart segmentation of both conventional (n = 20) and high-resolution (n = 45) CMRA (target) images, given segmented CTA (source) images for training. An additional supervised loss function was implemented to improve performance given 10%, 20% and 30% segmented CMRA cases. A fully supervised nn-UNet trained on the given CMRA segmentations was used as the benchmark.

RESULTS

The addition of a small number of segmented CMRA training cases substantially improved performance in both generative architectures in both standard and high-resolution datasets. Compared with the nn-UNet benchmark, the generative methods showed substantially better performance in the case of limited labelled cases. On the standard CMRA dataset, an average 12% (adversarial method) and 10% (variational method) improvement in Dice score was obtained.

CONCLUSIONS

Unsupervised domain-adaptation methods for CMRA segmentation can be boosted by the addition of a small number of supervised target training cases. When only few labelled cases are available, semi-supervised generative modelling is superior to supervised methods.

STAGAN: An approach for improve the stability of molecular graph generation based on generative adversarial networks

With the wide application of deep learning in Drug Discovery, deep generative model has shown its advantages in drug molecular generation. Generative adversarial networks can be used to learn the internal structure of molecules, but the training process may be unstable, such as gradient disappearance and model collapse, which may lead to the generation of molecules that do not conform to chemical rules or a single style. In this paper, a novel method called STAGAN was proposed to solve the difficulty of model training, by adding a new gradient penalty term in the discriminator and designing a parallel layer of batch normalization used in generator. As an illustration of method, STAGAN generated higher valid and unique molecules than previous models in training datasets from QM9 and ZINC-250K. This indicates that the proposed method can effectively solve the instability problem in the model training process, and can provide more instructive guidance for the further study of molecular graph generation.

Backdoor attack and defense in federated generative adversarial network-based medical image synthesis

Deep Learning-based image synthesis techniques have been applied in healthcare research for generating medical images to support open research and augment medical datasets. Training generative adversarial neural networks (GANs) usually require large amounts of training data. Federated learning (FL) provides a way of training a central model using distributed data while keeping raw data locally. However, given that the FL server cannot access the raw data, it is vulnerable to backdoor attacks, an adversarial by poisoning training data. Most backdoor attack strategies focus on classification models and centralized domains. It is still an open question if the existing backdoor attacks can affect GAN training and, if so, how to defend against the attack in the FL setting. In this work, we investigate the overlooked issue of backdoor attacks in federated GANs (FedGANs). The success of this attack is subsequently determined to be the result of some local discriminators overfitting the poisoned data and corrupting the local GAN equilibrium, which then further contaminates other clients when averaging the generator's parameters and yields high generator loss. Therefore, we proposed FedDetect, an efficient and effective way of defending against the backdoor attack in the FL setting, which allows the server to detect the client's adversarial behavior based on their losses and block the malicious clients. Our extensive experiments on two medical datasets with different modalities demonstrate the backdoor attack on FedGANs can result in synthetic images with low fidelity. After detecting and suppressing the detected malicious clients using the proposed defense strategy, we show that FedGANs can synthesize high-quality medical datasets (with labels) for data augmentation to improve classification models' performance.

A novel two-way rebalancing strategy for identifying carbonylation sites

BACKGROUND

As an irreversible post-translational modification, protein carbonylation is closely related to many diseases and aging. Protein carbonylation prediction for related patients is significant, which can help clinicians make appropriate therapeutic schemes. Because carbonylation sites can be used to indicate change or loss of protein function, integrating these protein carbonylation site data has been a promising method in prediction. Based on these protein carbonylation site data, some protein carbonylation prediction methods have been proposed. However, most data is highly class imbalanced, and the number of un-carbonylation sites greatly exceeds that of carbonylation sites. Unfortunately, existing methods have not addressed this issue adequately.

RESULTS

In this work, we propose a novel two-way rebalancing strategy based on the attention technique and generative adversarial network (Carsite_AGan) for identifying protein carbonylation sites. Specifically, Carsite_AGan proposes a novel undersampling method based on attention technology that allows sites with high importance value to be selected from un-carbonylation sites. The attention technique can obtain the value of each sample's importance. In the meanwhile, Carsite_AGan designs a generative adversarial network-based oversampling method to generate high-feasibility carbonylation sites. The generative adversarial network can generate high-feasibility samples through its generator and discriminator. Finally, we use a classifier like a nonlinear support vector machine to identify protein carbonylation sites.

CONCLUSIONS

Experimental results demonstrate that our approach significantly outperforms other resampling methods. Using our approach to resampling carbonylation data can significantly improve the effect of identifying protein carbonylation sites.

PVGAN: A Pathological Voice Generation Model Incorporating a Progressive Nesting Strategy

The voice generation task is to solve the problem of limited samples in the voice dataset using computer technology. By increasing the number of samples, the accuracy of voice disorder diagnosis can be improved, which has a wide range of application value in medical diagnosis and other fields. At present, there are insufficient models for detailed features such as pitch, timbre, and different frequency components in pathological voice data. Therefore, this paper proposes a PVGAN network for learning different frequency information of audio to generate pathological voice data. The proposed network captures the multi-scale features and different periodic patterns of audio signals by designing multiscale perceptual residual blocks and periodic discriminators. At the same time, a progressive nesting strategy was proposed to combine the generator and the discriminator to improve the learning ability of different resolution information. In addition, a latent mapping network is designed to fuse the latent vector with the condition information to generate sound features related to specific diseases or pathological states. The loss function is optimized to further improve the model performance. On the Saarbruecken Voice Database(SVD), the average values of each index of the data generated after training with different pathological types as conditional information are similar to the original data. Finally, the generated data were used to expand the SVD dataset, and the accuracy of the two classification experiments was improved to a certain extent.

A deep learning model for generating fundus autofluorescence images from color fundus photography

Background

Fundus Autofluorescence (FAF) is a valuable imaging technique used to assess metabolic alterations in the retinal pigment epithelium (RPE) associated with various age-related and disease-related changes. The practical uses of FAF are ever-growing. This study aimed to evaluate the effectiveness of a generative deep learning (DL) model in translating color fundus (CF) images into synthetic FAF images and explore its potential for enhancing screening of age-related macular degeneration (AMD).

Methods

A generative adversarial network (GAN) model was trained on pairs of CF and FAF images to generate synthetic FAF images. The quality of synthesized FAF images was assessed objectively by common generation metrics. Additionally, the clinical effectiveness of the generated FAF images in AMD classification was evaluated by measuring the area under the curve (AUC), using the LabelMe dataset.

Results

A total of 8410 FAF images from 2586 patients were analyzed. The synthesized FAF images exhibited an impressive objectively assessed quality, achieving a multi-scale structural similarity index (MS-SSIM) of 0.67. When evaluated on the LabelMe dataset, the combination of generated FAF images and CF images resulted in a noteworthy improvement in AMD classification accuracy, with the AUC increasing from 0.931 to 0.968.

Conclusions

This study presents the first attempt to use a generative deep learning model to create authentic and high-quality FAF images from CF images. The incorporation of the translated FAF images on top of CF images improved the accuracy of AMD classification. Overall, this study presents a promising approach to enhance large-scale AMD screening.

Two-stage generative adversarial networks for metal artifact reduction and visualization in ablation therapy of liver tumors

PURPOSE

The strong metal artifacts produced by the electrode needle cause poor image quality, thus preventing physicians from observing the surgical situation during the puncture process. To address this issue, we propose a metal artifact reduction and visualization framework for CT-guided ablation therapy of liver tumors.

METHODS

Our framework contains a metal artifact reduction model and an ablation therapy visualization model. A two-stage generative adversarial network is proposed to reduce the metal artifacts of intraoperative CT images and avoid image blurring. To visualize the puncture process, the axis and tip of the needle are localized, and then the needle is rebuilt in 3D space intraoperatively.

RESULTS

Experiments show that our proposed metal artifact reduction method achieves higher SSIM (0.891) and PSNR (26.920) values than the state-of-the-art methods. The accuracy of ablation needle reconstruction is 2.76 mm average in needle tip localization and 1.64° average in needle axis localization.

CONCLUSION

We propose a novel metal artifact reduction and an ablation therapy visualization framework for CT-guided ablation therapy of liver cancer. The experiment results indicate that our approach can reduce metal artifacts and improve image quality. Furthermore, our proposed method demonstrates the potential for displaying the relative position of the tumor and the needle intraoperatively.

Combining neural networks and image synthesis to enable automatic thoracic cavity segmentation of hyperpolarized 129Xe MRI without proton scans

RATIONALE AND OBJECTIVES

Quantification of 129Xe MRI relies on accurate segmentation of the thoracic cavity, typically performed manually using a combination of 1H and 129Xe scans. This can be accelerated by using Convolutional Neural Networks (CNNs) that segment only the 129Xe scan. However, this task is complicated by peripheral ventilation defects, which requires training CNNs with large, diverse datasets. Here, we accelerate the creation of training data by synthesizing 129Xe images with a variety of defects. We use this to train a 3D model to provide thoracic cavity segmentation from 129Xe ventilation MRI alone.

MATERIALS AND METHODS

Training and testing data consisted of 22 and 33 3D 129Xe ventilation images. Training data were expanded to 484 using Template-based augmentation while an additional 298 images were synthesized using the Pix2Pix model. This data was used to train both a 2D U-net and 3D V-net-based segmentation model using a combination of Dice-Focal and Anatomical Constraint loss functions. Segmentation performance was compared using Dice coefficients calculated over the entire lung and within ventilation defects.

RESULTS

Performance of both U-net and 3D segmentation was improved by including synthetic training data. The 3D models performed significantly better than U-net, and the 3D model trained with synthetic 129Xe images exhibited the highest overall Dice score of 0.929. Moreover, addition of synthetic training data improved the Dice score in ventilation defect regions from 0.545 to 0.588 for U-net and 0.739 to 0.765 for the 3D model.

CONCLUSION

It is feasible to obtain high-quality segmentations from 129Xe scan alone using 3D models trained with additional synthetic images.

Plane invariant segmentation of computed tomography images through weighted cross entropy optimized conditional GANs in compressed formats

Computed tomography (CT) scan provides first-hand knowledge to doctors to identify an ailment. Deep neural networks help enhance image understanding through segmentation and labeling. In this work, we implement two variants of Pix2Pix generative adversarial networks (GANs) with varying complexities of generator and discriminator networks for plane invariant segmentation of CT scan images and subsequently propose an effective generative adversarial network with a suitably weighted binary cross-entropy loss function followed by image processing layer necessary for getting high-quality output segmentation. Our conditional GAN is powered by a unique set of an encoder-decoder network that coupled with the image processing layer produces enhanced segmentation. The network can be extended to the complete set of Hounsfield units and can also be implemented on smartphones. Furthermore, we also demonstrate effects on accuracy, F-1 score, and Jaccard index by using the conditional GAN networks on the spine vertebrae dataset, thus achieving an average of 86.28 % accuracy, 90.5 % Jaccard index score, and 89.9 % F-1 score in predicting segmented maps for validation input images. In addition, an overall lifting of accuracy, F-1 score, and Jaccard index graph for validation images with better continuity has also been highlighted.

Automatic normalized digital color staining in the recognition of abnormal blood cells using generative adversarial networks

BACKGROUND AND OBJECTIVES

Combining knowledge of clinical pathologists and deep learning models is a growing trend in morphological analysis of cells circulating in blood to add objectivity, accuracy, and speed in diagnosing hematological and non-hematological diseases. However, the variability in staining protocols across different laboratories can affect the color of images and performance of automatic recognition models. The objective of this work is to develop, train and evaluate a new system for the normalization of color staining of peripheral blood cell images, so that it transforms images from different centers to map the color staining of a reference center (RC) while preserving the structural morphological features.

METHODS

The system has two modules, GAN1 and GAN2. GAN1 uses the PIX2PIX technique to fade original color images to an adaptive gray, while GAN2 transforms them into RGB normalized images. Both GANs have a similar structure, where the generator is a U-NET convolutional neural network with ResNet and the discriminator is a classifier with ResNet34 structure. Digitally stained images were evaluated using GAN metrics and histograms to assess the ability to modify color without altering cell morphology. The system was also evaluated as a pre-processing tool before cells undergo a classification process. For this purpose, a CNN classifier was designed for three classes: abnormal lymphocytes, blasts and reactive lymphocytes.

RESULTS

Training of all GANs and the classifier was performed using RC images, while evaluations were conducted using images from four other centers. Classification tests were performed before and after applying the stain normalization system. The overall accuracy reached a similar value around 96% in both cases for the RC images, indicating the neutrality of the normalization model for the reference images. On the contrary, it was a significant improvement in the classification performance when applying the stain normalization to the other centers. Reactive lymphocytes were the most sensitive to stain normalization, with true positive rates (TPR) increasing from 46.3% - 66% for the original images to 81.2% - 97.2% after digital staining. Abnormal lymphocytes TPR ranged from 31.9% - 95.7% with original images to 83% - 100% with digitally stained images. Blast class showed TPR ranges of 90.3% - 94.4% and 94.4% - 100%, for original and stained images, respectively.

CONCLUSIONS

The proposed GAN-based normalization staining approach improves the performance of classifiers with multicenter data sets by generating digitally stained images with a quality similar to the original images and adaptability to a reference staining standard. The system requires low computation cost and can help improve the performance of automatic recognition models in clinical settings.

Inflating 2D convolution weights for efficient generation of 3D medical images

BACKGROUND AND OBJECTIVE

The generation of three-dimensional (3D) medical images has great application potential since it takes into account the 3D anatomical structure. Two problems prevent effective training of a 3D medical generative model: (1) 3D medical images are expensive to acquire and annotate, resulting in an insufficient number of training images, and (2) a large number of parameters are involved in 3D convolution.

METHODS

We propose a novel GAN model called 3D Split&Shuffle-GAN. To address the 3D data scarcity issue, we first pre-train a two-dimensional (2D) GAN model using abundant image slices and inflate the 2D convolution weights to improve the initialization of the 3D GAN. Novel 3D network architectures are proposed for both the generator and discriminator of the GAN model to significantly reduce the number of parameters while maintaining the quality of image generation. Several weight inflation strategies and parameter-efficient 3D architectures are investigated.

RESULTS

Experiments on both heart (Stanford AIMI Coronary Calcium) and brain (Alzheimer's Disease Neuroimaging Initiative) datasets show that our method leads to improved 3D image generation quality (14.7 improvements on Frchet inception distance) with significantly fewer parameters (only 48.5% of the baseline method).

CONCLUSIONS

We built a parameter-efficient 3D medical image generation model. Due to the efficiency and effectiveness, it has the potential to generate high-quality 3D brain and heart images for real use cases.

Visual attribution using Adversarial Latent Transformations

The ability to accurately locate all indicators of disease within medical images is vital for comprehending the effects of the disease, as well as for weakly-supervised segmentation and localization of the diagnostic correlators of disease. Existing methods either use classifiers to make predictions based on class-salient regions or else use adversarial learning based image-to-image translation to capture such disease effects. However, the former does not capture all relevant features for visual attribution (VA) and are prone to data biases; the latter can generate adversarial (misleading) and inefficient solutions when dealing in pixel values. To address this issue, we propose a novel approach Visual Attribution using Adversarial Latent Transformations (VA2LT). Our method uses adversarial learning to generate counterfactual (CF) normal images from abnormal images by finding and modifying discrepancies in the latent space. We use cycle consistency between the query and CF latent representations to guide our training. We evaluate our method on three datasets including a synthetic dataset, the Alzheimer's Disease Neuroimaging Initiative dataset, and the BraTS dataset. Our method outperforms baseline and related methods on all datasets.

Generative adversarial networks in electrocardiogram synthesis: Recent developments and challenges

Training deep neural network classifiers for electrocardiograms (ECGs) requires sufficient data. However, imbalanced datasets pose a major problem for the training process and hence data augmentation is commonly performed. Generative adversarial networks (GANs) can create synthetic ECG data to augment such imbalanced datasets. This review aims at identifying the present literature concerning synthetic ECG signal generation using GANs to provide a comprehensive overview of architectures, quality evaluation metrics, and classification performances. Thirty publications from the years 2019 to 2022 were selected from three separate databases. Nine publications used a quality evaluation metric neglecting classification, eleven performed a classification but omitted a quality evaluation metric, and ten publications performed both. Twenty different quality evaluation metrics were observed. Overall, the classification performance of databases augmented with synthetically created ECG signals increased by 7 % to 98 % in accuracy and 6 % to 97 % in sensitivity. In conclusion, synthetic ECG signal generation using GANs represents a promising tool for data augmentation of imbalanced datasets. Consistent quality evaluation of generated signals remains challenging. Hence, future work should focus on the establishment of a gold standard for quality evaluation metrics for GANs.

Generative Adversarial Network (GAN) for Simulating Electroencephalography

Electroencephalographs record the electrical activity of your brain through the scalp. Electroencephalography is difficult to obtain due to its sensitivity and variability. Applications of electroencephalography such as for diagnosis, education, brain-computer interfaces require large samples of electroencephalography recording, however, it is often difficult to obtain the required datasets. Generative adversarial networks are robust deep learning framework which have proven themselves to be capable of synthesizing data. The robust nature of a generative adversarial network was used to generate multi-channel electroencephalography data in order to see if generative adversarial networks could reconstruct the spatio-temporal aspects of multi-channel electroencephalography signals. We were able to find that the synthetic electroencephalography data was able to replicate fine details of electroencephalography data and could potentially help us to generate large sample synthetic resting-state electroencephalography data for use in simulation testing of neuroimaging analyses. Generative adversarial networks (GANs) are robust deep-learning frameworks that can be trained to be convincing replicants of real data GANs were capable of generating "fake" EEG data that replicated fine details and topographies of "real" resting-state EEG data.

A brief introductory review to deep generative models for civil structural health monitoring

The use of deep generative models (DGMs) such as variational autoencoders, autoregressive models, flow-based models, energy-based models, generative adversarial networks, and diffusion models has been advantageous in various disciplines due to their high data generative skills. Using DGMs has become one of the most trending research topics in Artificial Intelligence in recent years. On the other hand, the research and development endeavors in the civil structural health monitoring (SHM) area have also been very progressive owing to the increasing use of Machine Learning techniques. As such, some of the DGMs have also been used in the civil SHM field lately. This short review communication paper aims to assist researchers in the civil SHM field in understanding the fundamentals of DGMs and, consequently, to help initiate their use for current and possible future engineering applications. On this basis, this study briefly introduces the concept and mechanism of different DGMs in a comparative fashion. While preparing this short review communication, it was observed that some DGMs had not been utilized or exploited fully in the SHM area. Accordingly, some representative studies presented in the civil SHM field that use DGMs are briefly overviewed. The study also presents a short comparative discussion on DGMs, their link to the SHM, and research directions.

High-Resolution 3D MRI With Deep Generative Networks via Novel Slice-Profile Transformation Super-Resolution

High-resolution magnetic resonance imaging (MRI) sequences, such as 3D turbo or fast spin-echo (TSE/FSE) imaging, are clinically desirable but suffer from long scanning time-related blurring when reformatted into preferred orientations. Instead, multi-slice two-dimensional (2D) TSE imaging is commonly used because of its high in-plane resolution but is limited clinically by poor through-plane resolution due to elongated voxels and the inability to generate multi-planar reformations due to staircase artifacts. Therefore, multiple 2D TSE scans are acquired in various orthogonal imaging planes, increasing the overall MRI scan time. In this study, we propose a novel slice-profile transformation super-resolution (SPTSR) framework with deep generative learning for through-plane super-resolution (SR) of multi-slice 2D TSE imaging. The deep generative networks were trained by synthesized low-resolution training input via slice-profile downsampling (SP-DS), and the trained networks inferred on the slice profile convolved (SP-conv) testing input for 5.5x through-plane SR. The network output was further slice-profile deconvolved (SP-deconv) to achieve an isotropic super-resolution. Compared to SMORE SR method and the networks trained by conventional downsampling, our SPTSR framework demonstrated the best overall image quality from 50 testing cases, evaluated by two abdominal radiologists. The quantitative analysis cross-validated the expert reader study results. 3D simulation experiments confirmed the quantitative improvement of the proposed SPTSR and the effectiveness of the SP-deconv step, compared to 3D ground-truths. Ablation studies were conducted on the individual contributions of SP-DS and SP-conv, networks structure, training dataset size, and different slice profiles.

Condition transfer between prestressed bridges using structural state translation for structural health monitoring

Implementing Structural Health Monitoring (SHM) systems with extensive sensing layouts on all civil structures is obviously expensive and unfeasible. Thus, estimating the state (condition) of dissimilar civil structures based on the information collected from other structures is regarded as a useful and essential way. For this purpose, Structural State Translation (SST) has been recently proposed to predict the response data of civil structures based on the information acquired from a dissimilar structure. This study uses the SST methodology to translate the state of one bridge (Bridge #1) to a new state based on the knowledge acquired from a structurally dissimilar bridge (Bridge #2). Specifically, the Domain-Generalized Cycle-Generative (DGCG) model is trained in the Domain Generalization learning approach on two distinct data domains obtained from Bridge #1; the bridges have two different conditions: State-H and State-D. Then, the model is used to generalize and transfer the knowledge on Bridge #1 to Bridge #2. In doing so, DGCG translates the state of Bridge #2 to the state that the model has learned after being trained. In one scenario, Bridge #2's State-H is translated to State-D; in another scenario, Bridge #2's State-D is translated to State-H. The translated bridge states are then compared with the real ones via modal identifiers and mean magnitude-squared coherence (MMSC), showing that the translated states are remarkably similar to the real ones. For instance, the modes of the translated and real bridge states are similar, with the maximum frequency difference of 1.12% and the minimum correlation of 0.923 in Modal Assurance Criterion values, as well as the minimum of 0.947 in Average MMSC values. In conclusion, this study demonstrates that SST is a promising methodology for research with data scarcity and population-based structural health monitoring (PBSHM). In addition, a critical discussion about the methodology adopted in this study is also offered to address some related concerns.

An Image Turing Test on Realistic Gastroscopy Images Generated by Using the Progressive Growing of Generative Adversarial Networks

Generative adversarial networks (GAN) in medicine are valuable techniques for augmenting unbalanced rare data, anomaly detection, and avoiding patient privacy issues. However, there were limits to generating high-quality endoscopic images with various characteristics, such as peristalsis, viewpoints, light sources, and mucous patterns. This study used the progressive growing of GAN (PGGAN) within the normal distribution dataset to confirm the ability to generate high-quality gastrointestinal images and investigated what barriers PGGAN has to generate endoscopic images. We trained the PGGAN with 107,060 gastroscopy images from 4165 normal patients to generate highly realistic 5122 pixel-sized images. For the evaluation, visual Turing tests were conducted on 100 real and 100 synthetic images to distinguish the authenticity of images by 19 endoscopists. The endoscopists were divided into three groups based on their years of clinical experience for subgroup analysis. The overall accuracy, sensitivity, and specificity of the 19 endoscopist groups were 61.3%, 70.3%, and 52.4%, respectively. The mean accuracy of the three endoscopist groups was 62.4 [Group I], 59.8 [Group II], and 59.1% [Group III], which was not considered a significant difference. There were no statistically significant differences in the location of the stomach. However, the real images with the anatomical landmark pylorus had higher detection sensitivity. The images generated by PGGAN showed highly realistic depictions that were difficult to distinguish, regardless of their expertise as endoscopists. However, it was necessary to establish GANs that could better represent the rugal folds and mucous membrane texture.

Generating synthetic clinical data that capture class imbalanced distributions with generative adversarial networks: Example using antiretroviral therapy for HIV

OBJECTIVE

Clinical data's confidential nature often limits the development of machine learning models in healthcare. Generative adversarial networks (GANs) can synthesise realistic datasets, but suffer from mode collapse, resulting in low diversity and bias towards majority demographics and common clinical practices. This work proposes an extension to the classic GAN framework that includes a variational autoencoder (VAE) and an external memory mechanism to overcome these limitations and generate synthetic data accurately describing imbalanced class distributions commonly found in clinical variables.

METHODS

The proposed method generated a synthetic dataset related to antiretroviral therapy for human immunodeficiency virus (ART for HIV). We evaluated it based on five metrics: (1) accurately representing imbalanced class distribution; (2) the realism of the individual variables; (3) the realism among variables; (4) patient disclosure risk; and (5) the utility of the generated dataset for developing downstream machine learning models.

RESULTS

The proposed method overcomes the issue of mode collapse and generates a synthetic dataset that accurately describes imbalanced class distributions commonly found in clinical variables. The generated data has a patient disclosure risk of 0.095%, lower than the 9% threshold stated by Health Canada and the European Medicines Agency, making it suitable for distribution to the research community with high security. The generated data also has high utility, indicating the potential of the proposed method to enable the development of downstream machine learning algorithms for healthcare applications using synthetic data.

CONCLUSION

Our proposed extension to the classic GAN framework, which includes a VAE and an external memory mechanism, represents a promising approach towards generating synthetic data that accurately describe imbalanced class distributions commonly found in clinical variables. This method overcomes the limitations of GANs and creates more realistic datasets with higher patient cohort diversity, facilitating the development of downstream machine learning algorithms for healthcare applications.

INTERPRETING GENERATIVE ADVERSARIAL NETWORKS TO INFER NATURAL SELECTION FROM GENETIC DATA

Understanding natural selection in humans and other species is a major focus for the use of machine learning in population genetics. Existing methods rely on computationally intensive simulated training data. Unlike efficient neutral coalescent simulations for demographic inference, realistic simulations of selection typically requires slow forward simulations. Because there are many possible modes of selection, a high dimensional parameter space must be explored, with no guarantee that the simulated models are close to the real processes. Mismatches between simulated training data and real test data can lead to incorrect inference. Finally, it is difficult to interpret trained neural networks, leading to a lack of understanding about what features contribute to classification. Here we develop a new approach to detect selection that requires relatively few selection simulations during training. We use a Generative Adversarial Network (GAN) trained to simulate realistic neutral data. The resulting GAN consists of a generator (fitted demographic model) and a discriminator (convolutional neural network). For a genomic region, the discriminator predicts whether it is "real" or "fake" in the sense that it could have been simulated by the generator. As the "real" training data includes regions that experienced selection and the generator cannot produce such regions, regions with a high probability of being real are likely to have experienced selection. To further incentivize this behavior, we "fine-tune" the discriminator with a small number of selection simulations. We show that this approach has high power to detect selection in simulations, and that it finds regions under selection identified by state-of-the art population genetic methods in three human populations. Finally, we show how to interpret the trained networks by clustering hidden units of the discriminator based on their correlation patterns with known summary statistics. In summary, our approach is a novel, efficient, and powerful way to use machine learning to detect natural selection.

Temporally consistent sequence-to-sequence translation of cataract surgeries

PURPOSE

Image-to-image translation methods can address the lack of diversity in publicly available cataract surgery data. However, applying image-to-image translation to videos-which are frequently used in medical downstream applications-induces artifacts. Additional spatio-temporal constraints are needed to produce realistic translations and improve the temporal consistency of translated image sequences.

METHODS

We introduce a motion-translation module that translates optical flows between domains to impose such constraints. We combine it with a shared latent space translation model to improve image quality. Evaluations are conducted regarding translated sequences' image quality and temporal consistency, where we propose novel quantitative metrics for the latter. Finally, the downstream task of surgical phase classification is evaluated when retraining it with additional synthetic translated data.

RESULTS

Our proposed method produces more consistent translations than state-of-the-art baselines. Moreover, it stays competitive in terms of the per-image translation quality. We further show the benefit of consistently translated cataract surgery sequences for improving the downstream task of surgical phase prediction.

CONCLUSION

The proposed module increases the temporal consistency of translated sequences. Furthermore, imposed temporal constraints increase the usability of translated data in downstream tasks. This allows overcoming some of the hurdles of surgical data acquisition and annotation and enables improving models' performance by translating between existing datasets of sequential frames.

Generative adversarial network based synthetic data training model for lightweight convolutional neural networks

Inadequate training data is a significant challenge for deep learning techniques, particularly in applications where data is difficult to get, and publicly available datasets are uncommon owing to ethical and privacy concerns. Various approaches, such as data augmentation and transfer learning, are employed to address this problem, which help to some extent in removing this limitation. However, after a certain amount of data augmentation, the quality of the generated data stalls, and transfer learning suffers from the issue of negative transfer. This paper proposes a novel generative adversarial network-based synthetic data training (GAN-ST) model to generate synthetic data for training a lightweight convolutional neural network (CNN). An enhanced generator is proposed to quickly saturate and cover the colour space of the training distribution. The GAN-ST model is based on Deep Convolutional Generative Adversarial Network(s) (DCGAN) and Conditional Generative Adversarial Network(s) (CGAN) models, which consist of an enhanced generator. The study evaluates the accuracy of a CNN model on the MNIST and CIFAR 10 datasets using both original and synthetic data. The results revealed an impressive classifier accuracy on the MNIST dataset, achieving an accuracy of 99.38% on GAN-ST-generated synthetic training data, which is only 0.05% lower than the performance on original data-based training. The classifier performance on the CIFAR dataset is also remarkable, achieving an accuracy of 90.23%. The performance of CNN trained using GAN-ST-based synthetic data is notable, with the most considerable improvement of 0.66% and 7.06%, over a single GAN-based synthetic data training for the MNIST and CIFAR datasets, respectively. By training two GANs independently, the GAN-ST model covers different parts of the original data distribution, resulting in a more diverse and realistic training data set for the classifier. This diverse set of synthetic data, when used to train a CNN, shows better generalization to new data, leading to improved classification accuracy.

Leak detection and localization in water distribution networks using conditional deep convolutional generative adversarial networks

This paper explores the use of 'conditional convolutional generative adversarial networks' (CDCGAN) for image-based leak detection and localization (LD&L) in water distribution networks (WDNs). The method employs pressure measurements and is based on four pillars: (1) hydraulic model-based generation of leak-free training data by taking into account the demand uncertainty, (2) conversion of hydraulic model input demand-output pressure pairs into images using kriging interpolation, (3) training of a CDCGAN model for image-to-image translation, and (4) using the structural similarity (SSIM) index for LD&L. SSIM, computed over the entire pressure distribution image is used for leak detection, and a local estimate of SSIM is employed for leak localization. The CDCGAN model employed in this paper is based on the pix2pix architecture. The effectiveness of the proposed methodology is demonstrated on leakage datasets under various scenarios. Results show that the method has an accuracy of approximately 70% for real-time leak detection. The proposed method is well-suited for real-time applications due to the low computational cost of CDCGAN predictions compared to WDN hydraulic models, is robust in presence of uncertainty due to the nature of generative adversarial networks, and scales well to large and variable-sized monitoring data due to the use of an image-based approach.

Generative adversarial network with radiomic feature reproducibility analysis for computed tomography denoising

BACKGROUND

Most computed tomography (CT) denoising algorithms have been evaluated using image quality analysis (IQA) methods developed for natural image, which do not adequately capture the texture details in medical imaging. Radiomics is an emerging image analysis technique that extracts texture information to provide a more objective basis for medical imaging diagnostics, overcoming the subjective nature of traditional methods. By utilizing the difficulty of reproducing radiomics features under different imaging protocols, we can more accurately evaluate the performance of CT denoising algorithms.

METHOD

We introduced radiomic feature reproducibility analysis as an evaluation metric for a denoising algorithm. Also, we proposed a low-dose CT denoising method based on a generative adversarial network (GAN), which outperformed well-known CT denoising methods.

RESULTS

Although the proposed model produced excellent results visually, the traditional image assessment metrics such as peak signal-to-noise ratio and structural similarity failed to show distinctive performance differences between the proposed method and the conventional ones. However, radiomic feature reproducibility analysis provided a distinctive assessment of the CT denoising performance. Furthermore, radiomic feature reproducibility analysis allowed fine-tuning of the hyper-parameters of the GAN.

CONCLUSION

We demonstrated that the well-tuned GAN architecture outperforms the well-known CT denoising methods. Our study is the first to introduce radiomics reproducibility analysis as an evaluation metric for CT denoising. We look forward that the study may bridge the gap between traditional objective and subjective evaluations in the clinical medical imaging field.

SuperstarGAN: Generative adversarial networks for image-to-image translation in large-scale domains

Image-to-image translation with generative adversarial networks (GANs) has been extensively studied in recent years. Among the models, StarGAN has achieved image-to-image translation for multiple domains with a single generator, whereas conventional models require multiple generators. However, StarGAN has several limitations, including the lack of capacity to learn mappings among large-scale domains; furthermore, StarGAN can barely express small feature changes. To address the limitations, we propose an improved StarGAN, namely SuperstarGAN. We adopted the idea, first proposed in controllable GAN (ControlGAN), of training an independent classifier with the data augmentation techniques to handle the overfitting problem in the classification of StarGAN structures. Since the generator with a well-trained classifier can express small features belonging to the target domain, SuperstarGAN achieves image-to-image translation in large-scale domains. Evaluated with a face image dataset, SuperstarGAN demonstrated improved performance in terms of Fréchet Inception distance (FID) and learned perceptual image patch similarity (LPIPS). Specifically, compared to StarGAN, SuperstarGAN exhibited decreased FID and LPIPS by 18.1% and 42.5%, respectively. Furthermore, we conducted an additional experiment with interpolated and extrapolated label values, indicating the ability of SuperstarGAN to control the degree of expression of the target domain features in generated images. Additionally, SuperstarGAN was successfully adapted to an animal face dataset and a painting dataset, where it can translate styles of animal faces (i.e., a cat to a tiger) and styles of painters (i.e., Hassam to Picasso), respectively, which explains the generality of SuperstarGAN regardless of datasets.

Model performance and interpretability of semi-supervised generative adversarial networks to predict oncogenic variants with unlabeled data

BACKGROUND

It remains an important challenge to predict the functional consequences or clinical impacts of genetic variants in human diseases, such as cancer. An increasing number of genetic variants in cancer have been discovered and documented in public databases such as COSMIC, but the vast majority of them have no functional or clinical annotations. Some databases, such as CiVIC are available with manual annotation of functional mutations, but the size of the database is small due to the use of human annotation. Since the unlabeled data (millions of variants) typically outnumber labeled data (thousands of variants), computational tools that take advantage of unlabeled data may improve prediction accuracy.

RESULT

To leverage unlabeled data to predict functional importance of genetic variants, we introduced a method using semi-supervised generative adversarial networks (SGAN), incorporating features from both labeled and unlabeled data. Our SGAN model incorporated features from clinical guidelines and predictive scores from other computational tools. We also performed comparative analysis to study factors that influence prediction accuracy, such as using different algorithms, types of features, and training sample size, to provide more insights into variant prioritization. We found that SGAN can achieve competitive performances with small labeled training samples by incorporating unlabeled samples, which is a unique advantage compared to traditional machine learning methods. We also found that manually curated samples can achieve a more stable predictive performance than publicly available datasets.

CONCLUSIONS

By incorporating much larger samples of unlabeled data, the SGAN method can improve the ability to detect novel oncogenic variants, compared to other machine-learning algorithms that use only labeled datasets. SGAN can be potentially used to predict the pathogenicity of more complex variants such as structural variants or non-coding variants, with the availability of more training samples and informative features.

Automatic generation of artificial images of leukocytes and leukemic cells using generative adversarial networks (syntheticcellgan)

BACKGROUND AND OBJECTIVES

Visual analysis of cell morphology has an important role in the diagnosis of hematological diseases. Morphological cell recognition is a challenge that requires experience and in-depth review by clinical pathologists. Within the new trend of introducing computer-aided diagnostic tools in laboratory medicine, models based on deep learning are being developed for the automatic identification of different types of cells in peripheral blood. In general, well-annotated large image sets are needed to train the models to reach a desired classification performance. This is especially relevant when it comes to discerning between cell images in which morphological differences are subtle and when it comes to low prevalent diseases with the consequent difficulty in collecting cell images. The objective of this work is to develop, train and validate SyntheticCellGAN (SCG), a new system for the automatic generation of artificial images of white blood cells, maintaining morphological characteristics very close to real cells found in practice in clinical laboratories.

METHODS

SCG is designed with two sequential generative adversarial networks. First, a Wasserstein structure is used to transform random noise vectors into low resolution images of basic mononuclear cells. Second, the concept of image-to-image translation is used to build specific models that transform the basic images into high-resolution final images with the realistic morphology of each cell type target: 1) the five groups of normal leukocytes (lymphocytes, monocytes, eosinophils, neutrophils and basophils); 2) atypical promyelocytes and hairy cells, which are two relevant cell types of complex morphology with low abundance in blood smears.

RESULTS

The images of the SCG system are evaluated with four experimental tests. In the first test we evaluated the generated images with quantitative metrics for GANs. In the second test, morphological verification of the artificial images is performed by expert clinical pathologists with 100% accuracy. In the third test, two classifiers based on convolutional neural networks (CNN) previously trained with images of real cells are used. Two sets of artificial images of the SCG system are classified with an accuracy of 95.36% and 94%, respectively. In the fourth test, three CNN classifiers are trained with artificial images of the SCG system. Real cells are identified with an accuracy ranging from 87.7% to 100%.

CONCLUSIONS

The SCG system has proven effective in creating images of all normal leukocytes and two low-prevalence cell classes associated with diseases such as acute promyelocyte leukemia and hairy cell leukemia. Once trained, the system requires low computational cost and can help augment high-quality image datasets to improve automatic recognition model training for clinical laboratory practice.

Synthetic MR image generation of macrotrabecular-massive hepatocellular carcinoma using generative adversarial networks

PURPOSE

The purpose of this study was to develop a method for generating synthetic MR images of macrotrabecular-massive hepatocellular carcinoma (MTM-HCC).

MATERIALS AND METHODS

A set of abdominal MR images including fat-saturated T1-weighted images obtained during the arterial and portal venous phases of enhancement and T2-weighted images of 91 patients with MTM-HCC, and another set of MR abdominal images from 67 other patients were used. Synthetic images were obtained using a 3-step pipeline that consisted in: (i), generating a synthetic MTM-HCC tumor on a neutral background; (ii), randomly selecting a background among the 67 patients and a position inside the liver; and (iii), merging the generated tumor in the background at the specified location. Synthetic images were qualitatively evaluated by three radiologists and quantitatively assessed using a mix of 1-nearest neighbor classifier metric and Fréchet inception distance.

RESULTS

A set of 1000 triplets of synthetic MTM-HCC images with consistent contrasts were successfully generated. Evaluation of selected synthetic images by three radiologists showed that the method gave realistic, consistent and diversified images. Qualitative and quantitative evaluation led to an overall score of 0.64.

CONCLUSION

This study shows the feasibility of generating realistic synthetic MR images with very few training data, by leveraging the wide availability of liver backgrounds. Further studies are needed to assess the added value of those synthetic images for automatic diagnosis of MTM-HCC.

Prediction of OCT images of short-term response to anti-VEGF treatment for diabetic macular edema using different generative adversarial networks

PURPOSE

This study sought to assess the predictive performance of optical coherence tomography (OCT) images for the response of diabetic macular edema (DME) patients to anti-vascular endothelial growth factor (VEGF) therapy generated from baseline images using generative adversarial networks (GANs).

METHODS

Patient information, including clinical and imaging data, was obtained from inpatients at the Ophthalmology Department of Qilu Hospital. 715 and 103 pairs of pre-and post-treatment OCT images of DME patients were included in the training and validation sets, respectively. The post-treatment OCT images were used to assess the validity of the generated images. Six different GAN models (CycleGAN, PairGAN, Pix2pixHD, RegGAN, SPADE, UNIT) were applied to predict the efficacy of anti-VEGF treatment by generating OCT images. Independent screening and evaluation experiments were conducted to validate the quality and comparability of images generated by different GAN models.

RESULTS

OCT images generated f GAN models exhibited high comparability to the real images, especially for edema absorption. RegGAN exhibited the highest prediction accuracy over the CycleGAN, PairGAN, Pix2pixHD, SPADE, and UNIT models. Further analyses were conducted based on the RegGAN. Most post-therapeutic OCT images (95/103) were difficult to differentiate from the real OCT images by retinal specialists. A mean absolute error of 26.74 ± 21.28 μm was observed for central macular thickness (CMT) between the synthetic and real OCT images.

CONCLUSION

Different generative adversarial networks have different prognostic efficacy for DME, and RegGAN yielded the best performance in our study. Different GAN models yielded good accuracy in predicting the OCT-based response to anti-VEGF treatment at one month. Overall, the application of GAN models can assist clinicians in prognosis prediction of patients with DME to design better treatment strategies and follow-up schedules.

Generative adversarial networks (GAN)-based data augmentation of rare liver cancers: The SFR 2021 Artificial Intelligence Data Challenge

PURPOSE

The 2021 edition of the Artificial Intelligence Data Challenge was organized by the French Society of Radiology together with the Centre National d'Études Spatiales and CentraleSupélec with the aim to implement generative adversarial networks (GANs) techniques to provide 1000 magnetic resonance imaging (MRI) cases of macrotrabecular-massive (MTM) hepatocellular carcinoma (HCC), a rare and aggressive subtype of HCC, generated from a limited number of real cases from multiple French centers.

MATERIALS AND METHODS

A dedicated platform was used by the seven inclusion centers to securely upload their anonymized MRI examinations including all three cross-sectional images (one late arterial and one portal-venous phase T1-weighted images and one fat-saturated T2-weighted image) in compliance with general data protection regulation. The quality of the database was checked by experts and manual delineation of the lesions was performed by the expert radiologists involved in each center. Multidisciplinary teams competed between October 11^th, 2021 and February 13^th, 2022.

RESULTS

A total of 91 MTM-HCC datasets of three images each were collected from seven French academic centers. Six teams with a total of 28 individuals participated in this challenge. Each participating team was asked to generate one thousand 3-image cases. The qualitative evaluation was performed by three radiologists using the Likert scale on ten randomly selected cases generated by each participant. A quantitative evaluation was also performed using two metrics, the Frechet inception distance and a leave-one-out accuracy of a 1-Nearest Neighbor algorithm.

CONCLUSION

This data challenge demonstrates the ability of GANs techniques to generate a large number of images from a small sample of imaging examinations of a rare malignant tumor.

A survey on deep learning applied to medical images: from simple artificial neural networks to generative models

Deep learning techniques, in particular generative models, have taken on great importance in medical image analysis. This paper surveys fundamental deep learning concepts related to medical image generation. It provides concise overviews of studies which use some of the latest state-of-the-art models from last years applied to medical images of different injured body areas or organs that have a disease associated with (e.g., brain tumor and COVID-19 lungs pneumonia). The motivation for this study is to offer a comprehensive overview of artificial neural networks (NNs) and deep generative models in medical imaging, so more groups and authors that are not familiar with deep learning take into consideration its use in medicine works. We review the use of generative models, such as generative adversarial networks and variational autoencoders, as techniques to achieve semantic segmentation, data augmentation, and better classification algorithms, among other purposes. In addition, a collection of widely used public medical datasets containing magnetic resonance (MR) images, computed tomography (CT) scans, and common pictures is presented. Finally, we feature a summary of the current state of generative models in medical image including key features, current challenges, and future research paths.

Biologically inspired robotic perception-action for soft fruit harvesting in vertical growing environments

Multiple interlinked factors like demographics, migration patterns, and economics are presently leading to the critical shortage of labour available for low-skilled, physically demanding tasks like soft fruit harvesting. This paper presents a biomimetic robotic solution covering the full 'Perception-Action' loop targeting harvesting of strawberries in a state-of-the-art vertical growing environment. The novelty emerges from both dealing with crop/environment variance as well as configuring the robot action system to deal with a range of runtime task constraints. Unlike the commonly used deep neural networks, the proposed perception system uses conditional Generative Adversarial Networks to identify the ripe fruit using synthetic data. The network can effectively train the synthetic data using the image-to-image translation concept, thereby avoiding the tedious work of collecting and labelling the real dataset. Once the harvest-ready fruit is localised using point cloud data generated by a stereo camera, our platform's action system can coordinate the arm to reach/cut the stem using the Passive Motion Paradigm framework inspired by studies on neural control of movement in the brain. Results from field trials for strawberry detection, reaching/cutting the stem of the fruit, and extension to analysing complex canopy structures/bimanual coordination (searching/picking) are presented. While this article focuses on strawberry harvesting, ongoing research towards adaptation of the architecture to other crops such as tomatoes and sweet peppers is briefly described.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11119-023-10000-4.