Attention Residual Learning for Skin Lesion Classification

Inter-Slice Context Residual Learning for 3D Medical Image Segmentation

Automated and accurate 3D medical image segmentation plays an essential role in assisting medical professionals to evaluate disease progresses and make fast therapeutic schedules. Although deep convolutional neural networks (DCNNs) have widely applied to this task, the accuracy of these models still need to be further improved mainly due to their limited ability to 3D context perception. In this paper, we propose the 3D context residual network (ConResNet) for the accurate segmentation of 3D medical images. This model consists of an encoder, a segmentation decoder, and a context residual decoder. We design the context residual module and use it to bridge both decoders at each scale. Each context residual module contains both context residual mapping and context attention mapping, the formal aims to explicitly learn the inter-slice context information and the latter uses such context as a kind of attention to boost the segmentation accuracy. We evaluated this model on the MICCAI 2018 Brain Tumor Segmentation (BraTS) dataset and NIH Pancreas Segmentation (Pancreas-CT) dataset. Our results not only demonstrate the effectiveness of the proposed 3D context residual learning scheme but also indicate that the proposed ConResNet is more accurate than six top-ranking methods in brain tumor segmentation and seven top-ranking methods in pancreas segmentation.

Automated Skin Lesion Segmentation Via an Adaptive Dual Attention Module

We present a convolutional neural network (CNN) equipped with a novel and efficient adaptive dual attention module (ADAM) for automated skin lesion segmentation from dermoscopic images, which is an essential yet challenging step for the development of a computer-assisted skin disease diagnosis system. The proposed ADAM has three compelling characteristics. First, we integrate two global context modeling mechanisms into the ADAM, one aiming at capturing the boundary continuity of skin lesion by global average pooling while the other dealing with the shape irregularity by pixel-wise correlation. In this regard, our network, thanks to the proposed ADAM, is capable of extracting more comprehensive and discriminative features for recognizing the boundary of skin lesions. Second, the proposed ADAM supports multi-scale resolution fusion, and hence can capture multi-scale features to further improve the segmentation accuracy. Third, as we harness a spatial information weighting method in the proposed network, our method can reduce a lot of redundancies compared with traditional CNNs. The proposed network is implemented based on a dual encoder architecture, which is able to enlarge the receptive field without greatly increasing the network parameters. In addition, we assign different dilation rates to different ADAMs so that it can adaptively capture distinguishing features according to the size of a lesion. We extensively evaluate the proposed method on both ISBI2017 and ISIC2018 datasets and the experimental results demonstrate that, without using network ensemble schemes, our method is capable of achieving better segmentation performance than state-of-the-art deep learning models, particularly those equipped with attention mechanisms.

Scale-Aware Transformers for Diagnosing Melanocytic Lesions

Diagnosing melanocytic lesions is one of the most challenging areas of pathology with extensive intra- and inter-observer variability. The gold standard for a diagnosis of invasive melanoma is the examination of histopathological whole slide skin biopsy images by an experienced dermatopathologist. Digitized whole slide images offer novel opportunities for computer programs to improve the diagnostic performance of pathologists. In order to automatically classify such images, representations that reflect the content and context of the input images are needed. In this paper, we introduce a novel self-attention-based network to learn representations from digital whole slide images of melanocytic skin lesions at multiple scales. Our model softly weighs representations from multiple scales, allowing it to discriminate between diagnosis-relevant and -irrelevant information automatically. Our experiments show that our method outperforms five other state-of-the-art whole slide image classification methods by a significant margin. Our method also achieves comparable performance to 187 practicing U.S. pathologists who interpreted the same cases in an independent study. To facilitate relevant research, full training and inference code is made publicly available at https://github.com/meredith-wenjunwu/ScATNet.

Self-channel-and-spatial-attention neural network for automated multi-organ segmentation on head and neck CT images

Accurate segmentation of organs at risk (OARs) is necessary for adaptive head and neck (H&N) cancer treatment planning, but manual delineation is tedious, slow, and inconsistent. A self-channel-and-spatial-attention neural network (SCSA-Net) is developed for H&N OAR segmentation on CT images. To simultaneously ease the training and improve the segmentation performance, the proposed SCSA-Net utilizes the self-attention ability of the network. Spatial and channel-wise attention learning mechanisms are both employed to adaptively force the network to emphasize the meaningful features and weaken the irrelevant features simultaneously. The proposed network was first evaluated on a public dataset, which includes 48 patients, then on a separate serial CT dataset, which contains ten patients who received weekly diagnostic fan-beam CT scans. On the second dataset, the accuracy of using SCSA-Net to track the parotid and submandibular gland volume changes during radiotherapy treatment was quantified. The Dice similarity coefficient (DSC), positive predictive value (PPV), sensitivity (SEN), average surface distance (ASD), and 95% maximum surface distance (95SD) were calculated on the brainstem, optic chiasm, optic nerves, mandible, parotid glands, and submandibular glands to evaluate the proposed SCSA-Net. The proposed SCSA-Net consistently outperforms the state-of-the-art methods on the public dataset. Specifically, compared with Res-Net and SE-Net, which is constructed from squeeze-and-excitation block equipped residual blocks, the DSC of the optic nerves and submandibular glands is improved by 0.06, 0.03 and 0.05, 0.04 by the SCSA-Net. Moreover, the proposed method achieves statistically significant improvements in terms of DSC on all and eight of nine OARs over Res-Net and SE-Net, respectively. The trained network was able to achieve good segmentation results on the serial dataset, but the results were further improved after fine-tuning of the model using the simulation CT images. For the parotids and submandibular glands, the volume changes of individual patients are highly consistent between the automated and manual segmentation (Pearson's correlation 0.97-0.99). The proposed SCSA-Net is computationally efficient to perform segmentation (sim 2 s/CT).

Skin Lesion Classification Using Densely Connected Convolutional Networks with Attention Residual Learning

Skin lesion classification is an effective approach aided by computer vision for the diagnosis of skin cancer. Though deep learning models presented advantages over traditional methods and brought tremendous breakthroughs, a precise diagnosis is still challenging because of the intra-class variation and inter-class similarity caused by the diversity of imaging methods and clinicopathology. In this paper, we propose a densely connected convolutional network with an attention and residual learning (ARDT-DenseNet) method for skin lesion classification. Each ARDT block consists of dense blocks, transition blocks and attention and residual modules. Compared to a residual network with the same number of convolutional layers, the size of the parameters of the densely connected network proposed in this paper has been reduced by half, while the accuracy of skin lesion classification is preserved. Our improved densely connected network adds an attention mechanism and residual learning after each dense block and transition block without introducing additional parameters. We evaluate the ARDT-DenseNet model with the ISIC 2016 and ISIC 2017 datasets. Our method achieves an ACC of 85.7% and an AUC of 83.7% in skin lesion classification with ISIC 2016 and an average AUC of 91.8% in skin lesion classification with ISIC 2017. The experimental results show that the method proposed in this paper has achieved a significant improvement in skin lesion classification, which is superior to that of the state-of-the-art method.

The effects of skin lesion segmentation on the performance of dermatoscopic image classification

BACKGROUND AND OBJECTIVE

Malignant melanoma (MM) is one of the deadliest types of skin cancer. Analysing dermatoscopic images plays an important role in the early detection of MM and other pigmented skin lesions. Among different computer-based methods, deep learning-based approaches and in particular convolutional neural networks have shown excellent classification and segmentation performances for dermatoscopic skin lesion images. These models can be trained end-to-end without requiring any hand-crafted features. However, the effect of using lesion segmentation information on classification performance has remained an open question.

METHODS

In this study, we explicitly investigated the impact of using skin lesion segmentation masks on the performance of dermatoscopic image classification. To do this, first, we developed a baseline classifier as the reference model without using any segmentation masks. Then, we used either manually or automatically created segmentation masks in both training and test phases in different scenarios and investigated the classification performances. The different scenarios included approaches that exploited the segmentation masks either for cropping of skin lesion images or removing the surrounding background or using the segmentation masks as an additional input channel for model training.

RESULTS

Evaluated on the ISIC 2017 challenge dataset which contained two binary classification tasks (i.e. MM vs. all and seborrheic keratosis (SK) vs. all) and based on the derived area under the receiver operating characteristic curve scores, we observed four main outcomes. Our results show that 1) using segmentation masks did not significantly improve the MM classification performance in any scenario, 2) in one of the scenarios (using segmentation masks for dilated cropping), SK classification performance was significantly improved, 3) removing all background information by the segmentation masks significantly degraded the overall classification performance, and 4) in case of using the appropriate scenario (using segmentation for dilated cropping), there is no significant difference of using manually or automatically created segmentation masks.

CONCLUSIONS

We systematically explored the effects of using image segmentation on the performance of dermatoscopic skin lesion classification.

Self-Supervised Learning to Increase the Performance of Skin Lesion Classification

To successfully train a deep neural network, a large amount of human-labeled data is required. Unfortunately, in many areas, collecting and labeling data is a difficult and tedious task. Several ways have been developed to mitigate the problem associated with the shortage of data, the most common of which is transfer learning. However, in many cases, the use of transfer learning as the only remedy is insufficient. In this study, we improve deep neural models training and increase the classification accuracy under a scarcity of data by the use of the self-supervised learning technique. Self-supervised learning allows an unlabeled dataset to be used for pretraining the network, as opposed to transfer learning that requires labeled datasets. The pretrained network can be then fine-tuned using the annotated data. Moreover, we investigated the effect of combining the self-supervised learning approach with transfer learning. It is shown that this strategy outperforms network training from scratch or with transfer learning. The tests were conducted on a very important and sensitive application (skin lesion classification), but the presented approach can be applied to a broader family of applications, especially in the medical domain where the scarcity of data is a real problem.

Automatic Classification System of Arrhythmias Using 12-Lead ECGs with a Deep Neural Network Based on an Attention Mechanism

Nowadays, a series of social problems caused by cardiovascular diseases are becoming increasingly serious. Accurate and efficient classification of arrhythmias according to an electrocardiogram is of positive significance for improving the health status of people all over the world. In this paper, a new neural network structure based on the most common 12-lead electrocardiograms was proposed to realize the classification of nine arrhythmias, which consists of Inception and GRU (Gated Recurrent Units) primarily. Moreover, a new attention mechanism is added to the model, which makes sense for data symmetry. The average F1 score obtained from three different test sets was over 0.886 and the highest was 0.919. The accuracy, sensitivity, and specificity obtained from the PhysioNet public database were 0.928, 0.901, and 0.984, respectively. As a whole, this deep neural network performed well in the multi-label classification of 12-lead ECG signals and showed better stability than other methods in the case of more test samples.

Semi-Supervised Medical Image Classification With Relation-Driven Self-Ensembling Model

Training deep neural networks usually requires a large amount of labeled data to obtain good performance. However, in medical image analysis, obtaining high-quality labels for the data is laborious and expensive, as accurately annotating medical images demands expertise knowledge of the clinicians. In this paper, we present a novel relation-driven semi-supervised framework for medical image classification. It is a consistency-based method which exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations, and leverages a self-ensembling model to produce high-quality consistency targets for the unlabeled data. Considering that human diagnosis often refers to previous analogous cases to make reliable decisions, we introduce a novel sample relation consistency (SRC) paradigm to effectively exploit unlabeled data by modeling the relationship information among different samples. Superior to existing consistency-based methods which simply enforce consistency of individual predictions, our framework explicitly enforces the consistency of semantic relation among different samples under perturbations, encouraging the model to explore extra semantic information from unlabeled data. We have conducted extensive experiments to evaluate our method on two public benchmark medical image classification datasets, i.e., skin lesion diagnosis with ISIC 2018 challenge and thorax disease classification with ChestX-ray14. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.

Triple attention learning for classification of 14 thoracic diseases using chest radiography

Chest X-ray is the most common radiology examinations for the diagnosis of thoracic diseases. However, due to the complexity of pathological abnormalities and lack of detailed annotation of those abnormalities, computer-aided diagnosis (CAD) of thoracic diseases remains challenging. In this paper, we propose the triple-attention learning (A ³ Net) model for this CAD task. This model uses the pre-trained DenseNet-121 as the backbone network for feature extraction, and integrates three attention modules in a unified framework for channel-wise, element-wise, and scale-wise attention learning. Specifically, the channel-wise attention prompts the deep model to emphasize the discriminative channels of feature maps; the element-wise attention enables the deep model to focus on the regions of pathological abnormalities; the scale-wise attention facilitates the deep model to recalibrate the feature maps at different scales. The proposed model has been evaluated on 112,120images in the ChestX-ray14 dataset with the official patient-level data split. Compared to state-of-the-art deep learning models, our model achieves the highest per-class AUC in classifying 13 out of 14 thoracic diseases and the highest average per-class AUC of 0.826 over 14 thoracic diseases.

Effective Melanoma Recognition Using Deep Convolutional Neural Network with Covariance Discriminant Loss

Melanoma recognition is challenging due to data imbalance and high intra-class variations and large inter-class similarity. Aiming at the issues, we propose a melanoma recognition method using deep convolutional neural network with covariance discriminant loss in dermoscopy images. Deep convolutional neural network is trained under the joint supervision of cross entropy loss and covariance discriminant loss, rectifying the model outputs and the extracted features simultaneously. Specifically, we design an embedding loss, namely covariance discriminant loss, which takes the first and second distance into account simultaneously for providing more constraints. By constraining the distance between hard samples and minority class center, the deep features of melanoma and non-melanoma can be separated effectively. To mine the hard samples, we also design the corresponding algorithm. Further, we analyze the relationship between the proposed loss and other losses. On the International Symposium on Biomedical Imaging (ISBI) 2018 Skin Lesion Analysis dataset, the two schemes in the proposed method can yield a sensitivity of 0.942 and 0.917, respectively. The comprehensive results have demonstrated the efficacy of the designed embedding loss and the proposed methodology.

Ψ-Net: Focusing on the border areas of intracerebral hemorrhage on CT images

BACKGROUND AND OBJECTIVE

The volume of the intracerebral hemorrhage (ICH) obtained from CT scans is essential for quantification and treatment planning. However,a fast and accurate volume acquisition brings great challenges. On the one hand, it is both time consuming and operator dependent for manual segmentation, which is the gold standard for volume estimation. On the other hand, low contrast to normal tissues, irregular shapes and distributions of the hemorrhage make the existing automatic segmentation methods hard to achieve satisfactory performance.

METHOD

To solve above problems, a CNN-based architecture is proposed in this work, consisting of a novel model, which is named as Ψ-Net and a multi-level training strategy. In the structure of Ψ-Net, a self-attention block and a contextual-attention block is designed to suppresses the irrelevant information and segment border areas of the hemorrhage more finely. Further, an multi-level training strategy is put forward to facilitate the training process. By adding the slice-level learning and a weighted loss, the multi-level training strategy effectively alleviates the problems of vanishing gradient and the class imbalance. The proposed training strategy could be applied to most of the segmentation networks, especially for complex models and on small datasets.

RESULTS

The proposed architecture is evaluated on a spontaneous ICH dataset and a traumatic ICH dataset. Compared to the previous works on the ICH sementation, the proposed architecture obtains the state-of-the-art performance(Dice of 0.950) on the spontaneous ICH, and comparable results(Dice of 0.895) with the best method on the traumatic ICH. On the other hand, the time consumption of the proposed architecture is much less than the previous methods on both training and inference. Morever, experiment results on various of models prove the universality of the multi-level training strategy.

CONCLUSIONS

This study proposed a novel CNN-based architecture, Ψ-Net with multi-level training strategy. It takes less time for training and achives superior performance than previous ICH segmentaion methods.

AMD-GAN: Attention encoder and multi-branch structure based generative adversarial networks for fundus disease detection from scanning laser ophthalmoscopy images

The scanning laser ophthalmoscopy (SLO) has become an important tool for the determination of peripheral retinal pathology, in recent years. However, the collected SLO images are easily interfered by the eyelash and frame of the devices, which heavily affect the key feature extraction of the images. To address this, we propose a generative adversarial network called AMD-GAN based on the attention encoder (AE) and multi-branch (MB) structure for fundus disease detection from SLO images. Specifically, the designed generator consists of two parts: the AE and generation flow network, where the real SLO images are encoded by the AE module to extract features and the generation flow network to handle the random Gaussian noise by a series of residual block with up-sampling (RU) operations to generate fake images with the same size as the real ones, where the AE is also used to mine features for generator. For discriminator, a ResNet network using MB is devised by copying the stage 3 and stage 4 structures of the ResNet-34 model to extract deep features. Furthermore, the depth-wise asymmetric dilated convolution is leveraged to extract local high-level contextual features and accelerate the training process. Besides, the last layer of discriminator is modified to build the classifier to detect the diseased and normal SLO images. In addition, the prior knowledge of experts is utilized to improve the detection results. Experimental results on the two local SLO datasets demonstrate that our proposed method is promising in detecting the diseased and normal SLO images with the experts labeling.

Fetal brain age estimation and anomaly detection using attention-based deep ensembles with uncertainty

MRI-based brain age prediction has been widely used to characterize normal brain development, and deviations from the typical developmental trajectory are indications of brain abnormalities. Age prediction of the fetal brain remains unexplored, although it can be of broad interest to prenatal examination given the limited diagnostic tools available for assessment of the fetal brain. In this work, we built an attention-based deep residual network based on routine clinical T2-weighted MR images of 659 fetal brains, which achieved an overall mean absolute error of 0.767 weeks and R² of 0.920 in fetal brain age prediction. The predictive uncertainty and estimation confidence were simultaneously quantified from the network as markers for detecting fetal brain anomalies using an ensemble method. The novel markers overcame the limitations in conventional brain age estimation and demonstrated promising diagnostic power in differentiating several types of fetal abnormalities, including small head circumference, malformations and ventriculomegaly with the area under the curve of 0.90, 0.90 and 0.67, respectively. In addition, attention maps were derived from the network, which revealed regional features that contributed to fetal age estimation at each gestational stage. The proposed attention-based deep ensembles demonstrated superior performance in fetal brain age estimation and fetal anomaly detection, which has the potential to be translated to prenatal diagnosis in clinical practice.

Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification

BACKGROUND AND OBJECTIVE

Skin cancer is among the most common cancer types in the white population and consequently computer aided methods for skin lesion classification based on dermoscopic images are of great interest. A promising approach for this uses transfer learning to adapt pre-trained convolutional neural networks (CNNs) for skin lesion diagnosis. Since pre-training commonly occurs with natural images of a fixed image resolution and these training images are usually significantly smaller than dermoscopic images, downsampling or cropping of skin lesion images is required. This however may result in a loss of useful medical information, while the ideal resizing or cropping factor of dermoscopic images for the fine-tuning process remains unknown.

METHODS

We investigate the effect of image size for skin lesion classification based on pre-trained CNNs and transfer learning. Dermoscopic images from the International Skin Imaging Collaboration (ISIC) skin lesion classification challenge datasets are either resized to or cropped at six different sizes ranging from 224 × 224 to 450 × 450. The resulting classification performance of three well established CNNs, namely EfficientNetB0, EfficientNetB1 and SeReNeXt-50 is explored. We also propose and evaluate a multi-scale multi-CNN (MSM-CNN) fusion approach based on a three-level ensemble strategy that utilises the three network architectures trained on cropped dermoscopic images of various scales.

RESULTS

Our results show that image cropping is a better strategy compared to image resizing delivering superior classification performance at all explored image scales. Moreover, fusing the results of all three fine-tuned networks using cropped images at all six scales in the proposed MSM-CNN approach boosts the classification performance compared to a single network or a single image scale. On the ISIC 2018 skin lesion classification challenge test set, our MSM-CNN algorithm yields a balanced multi-class accuracy of 86.2% making it the currently second ranked algorithm on the live leaderboard.

CONCLUSIONS

We confirm that the image size has an effect on skin lesion classification performance when employing transfer learning of CNNs. We also show that image cropping results in better performance compared to image resizing. Finally, a straightforward ensembling approach that fuses the results from images cropped at six scales and three fine-tuned CNNs is shown to lead to the best classification performance.

Prior-Attention Residual Learning for More Discriminative COVID-19 Screening in CT Images

We propose a conceptually simple framework for fast COVID-19 screening in 3D chest CT images. The framework can efficiently predict whether or not a CT scan contains pneumonia while simultaneously identifying pneumonia types between COVID-19 and Interstitial Lung Disease (ILD) caused by other viruses. In the proposed method, two 3D-ResNets are coupled together into a single model for the two above-mentioned tasks via a novel prior-attention strategy. We extend residual learning with the proposed prior-attention mechanism and design a new so-called prior-attention residual learning (PARL) block. The model can be easily built by stacking the PARL blocks and trained end-to-end using multi-task losses. More specifically, one 3D-ResNet branch is trained as a binary classifier using lung images with and without pneumonia so that it can highlight the lesion areas within the lungs. Simultaneously, inside the PARL blocks, prior-attention maps are generated from this branch and used to guide another branch to learn more discriminative representations for the pneumonia-type classification. Experimental results demonstrate that the proposed framework can significantly improve the performance of COVID-19 screening. Compared to other methods, it achieves a state-of-the-art result. Moreover, the proposed method can be easily extended to other similar clinical applications such as computer-aided detection and diagnosis of pulmonary nodules in CT images, glaucoma lesions in Retina fundus images, etc.

Automatic skin lesion classification based on mid-level feature learning

Dermoscopic images are widely used for melanoma detection. Many existing works based on traditional classification methods and deep learning models have been proposed for automatic skin lesion analysis. The traditional classification methods use hand-crafted features as input. However, due to the strong visual similarity between different classes of skin lesions and complex skin conditions, the hand-crafted features are not discriminative enough and fail in many cases. Recently, deep convolutional neural networks (CNN) have gained popularity since they can automatically learn optimal features during the training phase. Different from existing works, a novel mid-level feature learning method for skin lesion classification task is proposed in this paper. In this method, skin lesion segmentation is first performed to detect the regions of interest (ROI) of skin lesion images. Next, pretrained neural networks including ResNet and DenseNet are used as the feature extractors for the ROI images. Instead of using the extracted features directly as input of classifiers, the proposed method obtains the mid-level feature representations by utilizing the relationships among different image samples based on distance metric learning. The learned feature representation is a soft discriminative descriptor, having more tolerance to the hard samples and hence is more robust to the large intra-class difference and inter-class similarity. Experimental results demonstrate advantages of the proposed mid-level features, and the proposed method obtains state-of-the-art performance compared with the existing CNN based methods.

Deep Convolutional Neural Network for Melanoma Detection using Dermoscopy Images

Developing a fast and accurate classifier is an important part of a computer-aided diagnosis system for skin cancer. Melanoma is the most dangerous form of skin cancer which has a high mortality rate. Early detection and prognosis of melanoma can improve survival rates. In this paper, we propose a deep convolutional neural network for automated melanoma detection that is scalable to accommodate a variety of hardware and software constraints. Dermoscopic skin images collected from open sources were used for training the network. The trained network was then tested on a dataset of 2150 malignant or benign images. Overall, the classifier achieved high average values for accuracy, sensitivity, and specificity of 82.95%, 82.99%, and 83.89% respectively. It outperfomed other exisitng networks using the same dataset.

A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification

Automated skin lesion segmentation and classification are two most essential and related tasks in the computer-aided diagnosis of skin cancer. Despite their prevalence, deep learning models are usually designed for only one task, ignoring the potential benefits in jointly performing both tasks. In this paper, we propose the mutual bootstrapping deep convolutional neural networks (MB-DCNN) model for simultaneous skin lesion segmentation and classification. This model consists of a coarse segmentation network (coarse-SN), a mask-guided classification network (mask-CN), and an enhanced segmentation network (enhanced-SN). On one hand, the coarse-SN generates coarse lesion masks that provide a prior bootstrapping for mask-CN to help it locate and classify skin lesions accurately. On the other hand, the lesion localization maps produced by mask-CN are then fed into enhanced-SN, aiming to transfer the localization information learned by mask-CN to enhanced-SN for accurate lesion segmentation. In this way, both segmentation and classification networks mutually transfer knowledge between each other and facilitate each other in a bootstrapping way. Meanwhile, we also design a novel rank loss and jointly use it with the Dice loss in segmentation networks to address the issues caused by class imbalance and hard-easy pixel imbalance. We evaluate the proposed MB-DCNN model on the ISIC-2017 and PH2 datasets, and achieve a Jaccard index of 80.4% and 89.4% in skin lesion segmentation and an average AUC of 93.8% and 97.7% in skin lesion classification, which are superior to the performance of representative state-of-the-art skin lesion segmentation and classification methods. Our results suggest that it is possible to boost the performance of skin lesion segmentation and classification simultaneously via training a unified model to perform both tasks in a mutual bootstrapping way.

Semi-supervised WCE image classification with adaptive aggregated attention

Accurate abnormality classification in Wireless Capsule Endoscopy (WCE) images is crucial for early gastrointestinal (GI) tract cancer diagnosis and treatment, while it remains challenging due to the limited annotated dataset, the huge intra-class variances and the high degree of inter-class similarities. To tackle these dilemmas, we propose a novel semi-supervised learning method with Adaptive Aggregated Attention (AAA) module for automatic WCE image classification. Firstly, a novel deformation field based image preprocessing strategy is proposed to remove the black background and circular boundaries in WCE images. Then we propose a synergic network to learn discriminative image features, consisting of two branches: an abnormal regions estimator (the first branch) and an abnormal information distiller (the second branch). The first branch utilizes the proposed AAA module to capture global dependencies and incorporate context information to highlight the most meaningful regions, while the second branch mainly focuses on these calculated attention regions for accurate and robust abnormality classification. Finally, these two branches are jointly optimized by minimizing the proposed discriminative angular (DA) loss and Jensen-Shannon divergence (JS) loss with labeled data as well as unlabeled data. Comprehensive experiments have been conducted on the public CAD-CAP WCE dataset. The proposed method achieves 93.17% overall accuracy in a fourfold cross-validation, verifying its effectiveness for WCE image classification. The source code is available at https://github.com/Guo-Xiaoqing/SSL_WCE.

Attention by Selection: A Deep Selective Attention Approach to Breast Cancer Classification

Deep learning approaches are widely applied to histopathological image analysis due to the impressive levels of performance achieved. However, when dealing with high-resolution histopathological images, utilizing the original image as input to the deep learning model is computationally expensive, while resizing the original image to achieve low resolution incurs information loss. Some hard-attention based approaches have emerged to select possible lesion regions from images to avoid processing the original image. However, these hard-attention based approaches usually take a long time to converge with weak guidance, and valueless patches may be trained by the classifier. To overcome this problem, we propose a deep selective attention approach that aims to select valuable regions in the original images for classification. In our approach, a decision network is developed to decide where to crop and whether the cropped patch is necessary for classification. These selected patches are then trained by the classification network, which then provides feedback to the decision network to update its selection policy. With such a co-evolution training strategy, we show that our approach can achieve a fast convergence rate and high classification accuracy. Our approach is evaluated on a public breast cancer histopathological image database, where it demonstrates superior performance compared to state-of-the-art deep learning approaches, achieving approximately 98% classification accuracy while only taking 50% of the training time of the previous hard-attention approach.

Combining DC-GAN with ResNet for blood cell image classification

In medicine, white blood cells (WBCs) play an important role in the human immune system. The different types of WBC abnormalities are related to different diseases so that the total number and classification of WBCs are critical for clinical diagnosis and therapy. However, the traditional method of white blood cell classification is to segment the cells, extract features, and then classify them. Such method depends on the good segmentation, and the accuracy is not high. Moreover, the insufficient data or unbalanced samples can cause the low classification accuracy of model by using deep learning in medical diagnosis. To solve these problems, this paper proposes a new blood cell image classification framework which is based on a deep convolutional generative adversarial network (DC-GAN) and a residual neural network (ResNet). In particular, we introduce a new loss function which is improved the discriminative power of the deeply learned features. The experiments show that our model has a good performance on the classification of WBC images, and the accuracy reaches 91.7%. Graphical Abstract Overview of the proposed method, we use the deep convolution generative adversarial networks (DC-GAN) to generate new samples that are used as supplementary input to a ResNet, the transfer learning method is used to initialize the parameters of the network, the output of the DC-GAN and the parameters are applied the final classification network. In particular, we introduced a modified loss function for classification to increase inter-class variations and decrease intra-class differences.

Feature-shared adaptive-boost deep learning for invasiveness classification of pulmonary subsolid nodules in CT images

PURPOSE

In clinical practice, invasiveness is an important reference indicator for differentiating the malignant degree of subsolid pulmonary nodules. These nodules can be classified as atypical adenomatous hyperplasia (AAH), adenocarcinoma in situ (AIS), minimally invasive adenocarcinoma (MIA), or invasive adenocarcinoma (IAC). The automatic determination of a nodule's invasiveness based on chest CT scans can guide treatment planning. However, it is challenging, owing to the insufficiency of training data and their interclass similarity and intraclass variation. To address these challenges, we propose a two-stage deep learning strategy for this task: prior-feature learning followed by adaptive-boost deep learning.

METHODS

The adaptive-boost deep learning is proposed to train a strong classifier for invasiveness classification of subsolid nodules in chest CT images, using multiple 3D convolutional neural network (CNN)-based weak classifiers. Because ensembles of multiple deep 3D CNN models have a huge number of parameters and require large computing resources along with more training and testing time, the prior-feature learning is proposed to reduce the computations by sharing the CNN layers between all weak classifiers. Using this strategy, all weak classifiers can be integrated into a single network.

RESULTS

Tenfold cross validation of binary classification was conducted on a total of 1357 nodules, including 765 noninvasive (AAH and AIS) and 592 invasive nodules (MIA and IAC). Ablation experimental results indicated that the proposed binary classifier achieved an accuracy of 73.4 \% ± 1.4 with an AUC of 81.3 \% ± 2.2 . These results are superior compared to those achieved by three experienced chest imaging specialists who achieved an accuracy of 69.1 \% , 69.3 \% , and 67.9 \% , respectively. About 200 additional nodules were also collected. These nodules covered 50 cases for each category (AAH, AIS, MIA, and IAC, respectively). Both binary and multiple classifications were performed on these data and the results demonstrated that the proposed method definitely achieves better performance than the performance achieved by nonensemble deep learning methods.

CONCLUSIONS

It can be concluded that the proposed adaptive-boost deep learning can significantly improve the performance of invasiveness classification of pulmonary subsolid nodules in CT images, while the prior-feature learning can significantly reduce the total size of deep models. The promising results on clinical data show that the trained models can be used as an effective lung cancer screening tool in hospitals. Moreover, the proposed strategy can be easily extended to other similar classification tasks in 3D medical images.

A Novel LiDAR Data Classification Algorithm Combined CapsNet with ResNet

LiDAR data contain feature information such as the height and shape of the ground target and play an important role for land classification. The effect of convolutional neural network (CNN) for feature extraction on LiDAR data is very significant, however CNN cannot resolve the spatial relationship of features adequately. The capsule network (CapsNet) can identify the spatial variations of features and is widely used in supervised learning. In this article, the CapsNet is combined with the residual network (ResNet) to design a deep network-ResCapNet for improving the accuracy of LiDAR classification. The capsule network represents the features by vectors, which can account for the direction of the features and the relative position between the features. Therefore, more detailed feature information can be extracted. ResNet protects the integrity of information by passing input information to the output directly, which can solve the problem of network degradation caused by information loss in the traditional CNN propagation process to a certain extent. Two different LiDAR data sets and several classic machine learning algorithms are used for comparative experiments. The experimental results show that ResCapNet proposed in this article `improve the performance of LiDAR classification.

The Latest Trends in Attention Mechanisms and Their Application in Medical Imaging

딥러닝 기술은 빅데이터 및 컴퓨팅 파워를 기반으로 최근 영상의학 분야의 연구에서 괄목할만한 성과를 이루어 내고 있다. 하지만 성능 향상을 위해 딥러닝 네트워크가 깊어질수록 그 내부의 계산 과정을 해석하기 어려워졌는데, 이는 환자의 생명과 직결되는 의료분야의 의사 결정 과정에서는 매우 심각한 문제이다. 이를 해결하기 위해 “설명 가능한 인공지능 기술”이 연구되고 있으며, 그중 하나로 개발된 것이 바로 어텐션(attention) 기법이다. 본 종설에서는 이미 학습이 완료된 네트워크를 분석하기 위한 Post-hoc attention과, 네트워크 성능의 추가적인 향상을 위한 Trainable attention 두 종류의 기법에 대해 각각의 방법 및 의료 영상 연구에 적용된 사례, 그리고 향후 전망 등에 대해 자세히 다루고자 한다.

Thorax-Net: An Attention Regularized Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography

Deep learning techniques have been increasingly used to provide more accurate and more accessible diagnosis of thorax diseases on chest radiographs. However, due to the lack of dense annotation of large-scale chest radiograph data, this computer-aided diagnosis task is intrinsically a weakly supervised learning problem and remains challenging. In this paper, we propose a novel deep convolutional neural network called Thorax-Net to diagnose 14 thorax diseases using chest radiography. Thorax-Net consists of a classification branch and an attention branch. The classification branch serves as a uniform feature extraction-classification network to free users from the troublesome hand-crafted feature extraction and classifier construction. The attention branch exploits the correlation between class labels and the locations of pathological abnormalities via analyzing the feature maps learned by the classification branch. Feeding a chest radiograph to the trained Thorax-Net, a diagnosis is obtained by averaging and binarizing the outputs of two branches. The proposed Thorax-Net model has been evaluated against three state-of-the-art deep learning models using the patientwise official split of the ChestX-ray14 dataset and against other five deep learning models using the imagewise random data split. Our results show that Thorax-Net achieves an average per-class area under the receiver operating characteristic curve (AUC) of 0.7876 and 0.896 in both experiments, respectively, which are higher than the AUC values obtained by other deep models when they were all trained with no external data.

Semi-supervised adversarial model for benign-malignant lung nodule classification on chest CT

Classification of benign-malignant lung nodules on chest CT is the most critical step in the early detection of lung cancer and prolongation of patient survival. Despite their success in image classification, deep convolutional neural networks (DCNNs) always require a large number of labeled training data, which are not available for most medical image analysis applications due to the work required in image acquisition and particularly image annotation. In this paper, we propose a semi-supervised adversarial classification (SSAC) model that can be trained by using both labeled and unlabeled data for benign-malignant lung nodule classification. This model consists of an adversarial autoencoder-based unsupervised reconstruction network R, a supervised classification network C, and learnable transition layers that enable the adaption of the image representation ability learned by R to C. The SSAC model has been extended to the multi-view knowledge-based collaborative learning, aiming to employ three SSACs to characterize each nodule's overall appearance, heterogeneity in shape and texture, respectively, and to perform such characterization on nine planar views. The MK-SSAC model has been evaluated on the benchmark LIDC-IDRI dataset and achieves an accuracy of 92.53% and an AUC of 95.81%, which are superior to the performance of other lung nodule classification and semi-supervised learning approaches.