Active Appearance Model Induced Generative Adversarial Network for Controlled Data Augmentation

A novel image augmentation based on statistical shape and intensity models: application to the segmentation of hip bones from CT images

BACKGROUND

The collection and annotation of medical images are hindered by data scarcity, privacy, and ethical reasons or limited resources, negatively affecting deep learning approaches. Data augmentation is often used to mitigate this problem, by generating synthetic images from training sets to improve the efficiency and generalization of deep learning models.

METHODS

We propose the novel use of statistical shape and intensity models (SSIM) to generate augmented images with variety in both shape and intensity of imaged structures and surroundings. The SSIM uses segmentations from training images to create co-registered tetrahedral meshes of the structures and to efficiently encode image intensity in their interior with Bernstein polynomials. In the context of segmentation of hip joint (pathological) bones from retrospective computed tomography images of 232 patients, we compared the impact of SSIM-based and basic augmentations on the performance of a U-Net model.

RESULTS

In a fivefold cross-validation, the SSIM augmentation improved segmentation robustness and accuracy. In particular, the combination of basic and SSIM augmentation outperformed trained models not using any augmentation, or relying exclusively on a simple form of augmentation, achieving Dice similarity coefficient and Hausdorff distance of 0.95 [0.93-0.96] and 6.16 [4.90-8.08] mm (median [25th-75th percentiles]), comparable to previous work on pathological hip segmentation.

CONCLUSIONS

We proposed a novel augmentation varying both the shape and appearance of structures in generated images. Tested on bone segmentation, our approach is generalizable to other structures or tasks such as classification, as long as SSIM can be built from training data.

RELEVANCE STATEMENT

Our data augmentation approach produces realistic shape and appearance variations of structures in generated images, which supports the clinical adoption of AI in radiology by alleviating the collection of clinical imaging data and by improving the performance of AI applications.

KEY POINTS

• Data augmentation generally improves the accuracy and generalization of deep learning models. • Traditional data augmentation does not consider the appearance of imaged structures. • Statistical shape and intensity models (SSIM) synthetically generate variations of imaged structures. • SSIM support novel augmentation approaches, demonstrated with computed tomography bone segmentation.

A multitask dual-stream attention network for the identification of KRAS mutation in colorectal cancer

PURPOSE

It is of great significance to accurately identify the KRAS gene mutation status for patients in tumor prognosis and personalized treatment. Although the computer-aided diagnosis system based on deep learning has gotten all-round development, its performance still cannot meet the current clinical application requirements due to the inherent limitations of small-scale medical image data set and inaccurate lesion feature extraction. Therefore, our aim is to propose a deep learning model based on T2 MRI of colorectal cancer (CRC) patients to identify whether KRAS gene is mutated.

METHODS

In this research, a multitask attentive model is proposed to identify KRAS gene mutations in patients, which is mainly composed of a segmentation subnetwork and an identification subnetwork. Specifically, at first, the features extracted by the encoder of segmentation model are used as guidance information to guide the two attention modules in the identification network for precise activation of the lesion area. Then the original image of the lesion and the segmentation result are concatenated for feature extraction. Finally, features extracted from the second step are combined with features activated by the attention modules to identify the gene mutation status. In this process, we introduce the interlayer loss function to encourage the similarity of the two subnetwork parameters and ensure that the key features are fully extracted to alleviate the overfitting problem caused by small data set to some extent.

RESULTS

The proposed identification model is benchmarked primarily using 15-fold cross validation. Three hundred and eighty-two images from 36 clinical cases were used to test the model. For the identification of KRAS mutation status, the average accuracy is 89.95 ± 1.23%, the average sensitivity is 89.29 ± 1.79%, the average specificity is 90.53 ± 2.45%, and the average area under the curve (AUC) is 95.73 ± 0.52%. For segmentation of lesions, the average dice is 88.11 ± 0.86%.

CONCLUSIONS

We developed a novel deep learning-based model to identify the KRAS status in CRC. We demonstrated the excellent properties of the proposed identification through comparison with ground truth gene mutation status of 36 clinical cases. And all these results show that the novel method has great potential for clinical application.

Automated Iterative Label Transfer Improves Segmentation of Noisy Cells in Adaptive Optics Retinal Images

High quality data labeling is essential for improving the accuracy of deep learning applications in medical imaging. However, noisy images are not only under-represented in training datasets, but also, labeling of noisy data is low quality. Unfortunately, noisy images with poor quality labels are exacerbated by traditional data augmentation strategies. Real world images contain noise and can lead to unexpected drops in algorithm performance. In this paper, we present a non-traditional, purposeful data augmentation method to specifically transfer high quality automated labels into noisy image regions for incorporation into the training dataset. The overall approach is based on the use of paired images of the same cells in which variable image noise results in cell segmentation failures. Iteratively updating the cell segmentation model with accurate labels of noisy image areas resulted in an improvement in Dice coefficient from 77% to 86%. This was achieved by adding only 3.4% more cells to the training dataset, showing that local label transfer through graph matching is an effective augmentation strategy to improve segmentation.

Active Cell Appearance Model Induced Generative Adversarial Networks for Annotation-Efficient Cell Segmentation and Identification on Adaptive Optics Retinal Images

Data annotation is a fundamental precursor for establishing large training sets to effectively apply deep learning methods to medical image analysis. For cell segmentation, obtaining high quality annotations is an expensive process that usually requires manual grading by experts. This work introduces an approach to efficiently generate annotated images, called "A-GANs", created by combining an active cell appearance model (ACAM) with conditional generative adversarial networks (C-GANs). ACAM is a statistical model that captures a realistic range of cell characteristics and is used to ensure that the image statistics of generated cells are guided by real data. C-GANs utilize cell contours generated by ACAM to produce cells that match input contours. By pairing ACAM-generated contours with A-GANs-based generated images, high quality annotated images can be efficiently generated. Experimental results on adaptive optics (AO) retinal images showed that A-GANs robustly synthesize realistic, artificial images whose cell distributions are exquisitely specified by ACAM. The cell segmentation performance using as few as 64 manually-annotated real AO images combined with 248 artificially-generated images from A-GANs was similar to the case of using 248 manually-annotated real images alone (Dice coefficients of 88% for both). Finally, application to rare diseases in which images exhibit never-seen characteristics demonstrated improvements in cell segmentation without the need for incorporating manual annotations from these new retinal images. Overall, A-GANs introduce a methodology for generating high quality annotated data that statistically captures the characteristics of any desired dataset and can be used to more efficiently train deep-learning-based medical image analysis applications.

A Progressive Generative Adversarial Method for Structurally Inadequate Medical Image Data Augmentation

The generation-based data augmentation method can overcome the challenge caused by the imbalance of medical image data to a certain extent. However, most of the current research focus on images with unified structure which are easy to learn. What is different is that ultrasound images are structurally inadequate, making it difficult for the structure to be captured by the generative network, resulting in the generated image lacks structural legitimacy. Therefore, a Progressive Generative Adversarial Method for Structurally Inadequate Medical Image Data Augmentation is proposed in this paper, including a network and a strategy. Our Progressive Texture Generative Adversarial Network alleviates the adverse effect of completely truncating the reconstruction of structure and texture during the generation process and enhances the implicit association between structure and texture. The Image Data Augmentation Strategy based on Mask-Reconstruction overcomes data imbalance from a novel perspective, maintains the legitimacy of the structure in the generated data, as well as increases the diversity of disease data interpretably. The experiments prove the effectiveness of our method on data augmentation and image reconstruction on Structurally Inadequate Medical Image both qualitatively and quantitatively. Finally, the weakly supervised segmentation of the lesion is the additional contribution of our method.

Spatially Aware Dense-LinkNet Based Regression Improves Fluorescent Cell Detection in Adaptive Optics Ophthalmic Images

Retinal pigment epithelial (RPE) cells play an important role in nourishing retinal neurosensory photoreceptor cells, and numerous blinding diseases are associated with RPE defects. Their fluorescence signature can now be visualized in the living human eye using adaptive optics (AO) imaging combined with indocyanine green (ICG), which motivates us to develop an automated RPE detection method to improve the quantitative evaluation of RPE status in patients. This paper proposes a spatially-aware, Dense-LinkNet-based regression approach to improve the detection of in vivo fluorescent cell patterns, achieving precision, recall, and F1-Score of 93.6 ± 4.3%, 81.4 ± 9.5%, and 86.7 ± 5.7%, respectively. These results demonstrate the utility of incorporating spatial inputs into a deep learning-based regression framework for cell detection.

Graded Image Generation Using Stratified CycleGAN

In medical imaging, CycleGAN has been used for various image generation tasks, including image synthesis, image denoising, and data augmentation. However, when pushing the technical limits of medical imaging, there can be a substantial variation in image quality. Here, we demonstrate that images generated by CycleGAN can be improved through explicit grading of image quality, which we call stratified CycleGAN. In this image generation task, CycleGAN is used to upgrade the image quality and content of near-infrared fluorescent (NIRF) retinal images. After manual assignment of grading scores to a small subset of the data, semi-supervised learning is applied to propagate grades across the remainder of the data and set up the training data. These scores are embedded into the CycleGAN by adding the grading score as a conditional input to the generator and by integrating an image quality classifier into the discriminator. We validate the efficacy of the proposed stratified CycleGAN by considering pairs of NIRF images at the same retinal regions (imaged with and without correction of optical aberrations achieved using adaptive optics), with the goal being to restore image quality in aberrated images such that cellular-level detail can be obtained. Overall, stratified CycleGAN generated higher quality synthetic images than traditional CycleGAN. Evaluation of cell detection accuracy confirmed that synthetic images were faithful to ground truth images of the same cells. Across this challenging dataset, F1-score improved from 76.9 ± 5.7% when using traditional CycleGAN to 85.0±3.4% when using stratified CycleGAN. These findings demonstrate the potential of stratified Cycle-GAN to improve the synthesis of medical images that exhibit a graded variation in image quality.