Attention Residual Learning for Skin Lesion Classification

Deep Learning Methods for Accurate Skin Cancer Recognition and Mobile Application

Although many efforts have been made through past years, skin cancer recognition from medical images is still an active area of research aiming at more accurate results. Many efforts have been made in recent years based on deep learning neural networks. Only a few, however, are based on a single deep learning model and targeted to create a mobile application. Contributing to both efforts, first we present a summary of the required medical knowledge on skin cancer, followed by an extensive summary of the most recent related works. Afterwards, we present 11 CNN (convolutional neural network) candidate single architectures. We train and test those 11 CNN architectures, using the HAM10000 dataset, concerning seven skin lesion classes. To face the imbalance problem and the high similarity between images of some skin lesions, we apply data augmentation (during training), transfer learning and fine-tuning. From the 11 CNN architecture configurations, DenseNet169 produced the best results. It achieved an accuracy of 92.25%, a recall (sensitivity) of 93.59% and an F1-score of 93.27%, which outperforms existing state-of-the-art efforts. We used a light version of DenseNet169 in constructing a mobile android application, which was mapped as a two-class model (benign or malignant). A picture is taken via the mobile device camera, and after manual cropping, it is classified into benign or malignant type. The application can also inform the user about the allowed sun exposition time based on the current UV radiation degree, the phototype of the user’s skin and the degree of the used sunscreen. In conclusion, we achieved state-of-the-art results in skin cancer recognition based on a single, relatively light deep learning model, which we also used in a mobile application.

A Novel Hybrid Deep Learning Approach for Skin Lesion Segmentation and Classification

Skin cancer is one of the most common diseases that can be initially detected by visual observation and further with the help of dermoscopic analysis and other tests. As at an initial stage, visual observation gives the opportunity of utilizing artificial intelligence to intercept the different skin images, so several skin lesion classification methods using deep learning based on convolution neural network (CNN) and annotated skin photos exhibit improved results. In this respect, the paper presents a reliable approach for diagnosing skin cancer utilizing dermoscopy images in order to improve health care professionals' visual perception and diagnostic abilities to discriminate benign from malignant lesions. The swarm intelligence (SI) algorithms were used for skin lesion region of interest (RoI) segmentation from dermoscopy images, and the speeded-up robust features (SURF) was used for feature extraction of the RoI marked as the best segmentation result obtained using the Grasshopper Optimization Algorithm (GOA). The skin lesions are classified into two groups using CNN against three data sets, namely, ISIC-2017, ISIC-2018, and PH-2 data sets. The proposed segmentation and classification techniques' results are assessed in terms of classification accuracy, sensitivity, specificity, F-measure, precision, MCC, dice coefficient, and Jaccard index, with an average classification accuracy of 98.42 percent, precision of 97.73 percent, and MCC of 0.9704 percent. In every performance measure, our suggested strategy exceeds previous work.

Explainable artificial intelligence in skin cancer recognition: A systematic review

BACKGROUND

Due to their ability to solve complex problems, deep neural networks (DNNs) are becoming increasingly popular in medical applications. However, decision-making by such algorithms is essentially a black-box process that renders it difficult for physicians to judge whether the decisions are reliable. The use of explainable artificial intelligence (XAI) is often suggested as a solution to this problem. We investigate how XAI is used for skin cancer detection: how is it used during the development of new DNNs? What kinds of visualisations are commonly used? Are there systematic evaluations of XAI with dermatologists or dermatopathologists?

METHODS

Google Scholar, PubMed, IEEE Explore, Science Direct and Scopus were searched for peer-reviewed studies published between January 2017 and October 2021 applying XAI to dermatological images: the search terms histopathological image, whole-slide image, clinical image, dermoscopic image, skin, dermatology, explainable, interpretable and XAI were used in various combinations. Only studies concerned with skin cancer were included.

RESULTS

37 publications fulfilled our inclusion criteria. Most studies (19/37) simply applied existing XAI methods to their classifier to interpret its decision-making. Some studies (4/37) proposed new XAI methods or improved upon existing techniques. 14/37 studies addressed specific questions such as bias detection and impact of XAI on man-machine-interactions. However, only three of them evaluated the performance and confidence of humans using CAD systems with XAI.

CONCLUSION

XAI is commonly applied during the development of DNNs for skin cancer detection. However, a systematic and rigorous evaluation of its usefulness in this scenario is lacking.

Hierarchy-aware contrastive learning with late fusion for skin lesion classification

BACKGROUND AND OBJECTIVE

The incidence rate of skin cancers is increasing worldwide annually. Using machine learning and deep learning for skin lesion classification is one of the essential research topics. In this study, we formulate a major-type misclassification problem that previous studies did not consider in the multi-class skin lesion classification. Moreover, addressing the major-type misclassification problem is significant for real-world computer-aided diagnosis.

METHODS

This study presents a novel method, namely Hierarchy-Aware Contrastive Learning with Late Fusion (HAC-LF), to improve the overall performance of multi-class skin classification. In HAC-LF, we design a new loss function, Hierarchy-Aware Contrastive Loss (HAC Loss), to reduce the impact of the major-type misclassification problem. The late fusion method is applied to balance the major-type and multi-class classification performance.

RESULTS

We conduct a series of experiments with the ISIC 2019 Challenges dataset, which consists of three skin lesion datasets, to verify the performance of our methods. The results show that our proposed method surpasses the representative deep learning methods for skin lesion classification in all evaluation metrics used in this study. HAC-LF achieves 0.871, 0.842, 0.889 for accuracy, sensitivity, and specificity in the major-type classification, respectively. With the imbalanced class distribution, HAC-LF outperforms the baseline model regarding the sensitivity of minority classes.

CONCLUSIONS

This research formulates a major-type misclassification problem. We propose HAC-LF to deal with it and boost the multi-class skin lesion classification performance. According to the results, the advantage of HAC-LF is that the proposed HAC Loss can beneficially reduce the impact of the major-type misclassification by decreasing the major-type error rate. Besides the medical field HAC-LF is promising to be applied to other domains possessing the data with the hierarchical structure.

ReconResNet: Regularised residual learning for MR image reconstruction of Undersampled Cartesian and Radial data

MRI is an inherently slow process, which leads to long scan time for high-resolution imaging. The speed of acquisition can be increased by ignoring parts of the data (undersampling). Consequently, this leads to the degradation of image quality, such as loss of resolution or introduction of image artefacts. This work aims to reconstruct highly undersampled Cartesian or radial MR acquisitions, with better resolution and with less to no artefact compared to conventional techniques like compressed sensing. In recent times, deep learning has emerged as a very important area of research and has shown immense potential in solving inverse problems, e.g. MR image reconstruction. In this paper, a deep learning based MR image reconstruction framework is proposed, which includes a modified regularised version of ResNet as the network backbone to remove artefacts from the undersampled image, followed by data consistency steps that fusions the network output with the data already available from undersampled k-space in order to further improve reconstruction quality. The performance of this framework for various undersampling patterns has also been tested, and it has been observed that the framework is robust to deal with various sampling patterns, even when mixed together while training, and results in very high quality reconstruction, in terms of high SSIM (highest being 0.990 ± 0.006 for acceleration factor of 3.5), while being compared with the fully sampled reconstruction. It has been shown that the proposed framework can successfully reconstruct even for an acceleration factor of 20 for Cartesian (0.968 ± 0.005) and 17 for radially (0.962 ± 0.012) sampled data. Furthermore, it has been shown that the framework preserves brain pathology during reconstruction while being trained on healthy subjects.

Handling Imbalanced Data: Uncertainty-guided Virtual Adversarial Training with Batch Nuclear-norm Optimization for Semi-supervised Medical Image Classification

In many clinical settings, a lot of medical image datasets suffer from imbalance problems, which makes predictions of trained models to be biased toward majority classes. Semi-supervised Learning (SSL) algorithms trained with such imbalanced datasets become more problematic since pseudo-supervision of unlabeled data are generated from the model's biased predictions. To address these issues, in this work, we propose a novel semi-supervised deep learning method, i.e., uncertainty-guided virtual adversarial training (VAT) with batch nuclear-norm (BNN) optimization, for large-scale medical image classification. To effectively exploit useful information from both labeled and unlabeled data, we leverage VAT and BNN optimization to harness the underlying knowledge, which helps to improve discriminability, diversity and generalization of the trained models. More concretely, our network is trained by minimizing a combination of four types of losses, including a supervised cross-entropy loss, a BNN loss defined on the output matrix of labeled data batch (lBNN loss), a negative BNN loss defined on the output matrix of unlabeled data batch (uBNN loss), and a VAT loss on both labeled and unlabeled data. We additionally propose to use uncertainty estimation to filter out unlabeled samples near the decision boundary when computing the VAT loss. We conduct comprehensive experiments to evaluate the performance of our method on two publicly available datasets and one in-house collected dataset. The experimental results demonstrated that our method achieved better results than state-of-the-art SSL methods.

COVID detection from Chest X-Ray Images using multi-scale attention

Deep learning based methods have shown great promise in achieving accurate automatic detection of Coronavirus Disease (COVID) - 19 from Chest X-Ray (CXR) images. However, incorporating explainability in these solutions remains relatively less explored. We present a hierarchical classification approach for separating normal, non-COVID pneumonia (NCP) and COVID cases using CXR images. We demonstrate that the proposed method achieves clinically consistent explanations. We achieve this using a novel multi-scale attention architecture called Multi-scale Attention Residual Learning (MARL) and a new loss function based on conicity for training the proposed architecture. The proposed classification strategy has two stages. The first stage uses a model derived from DenseNet to separate pneumonia cases from normal cases while the second stage uses the MARL architecture to discriminate between COVID and NCP cases. With a five-fold cross validation, the proposed method achieves 93%, 96.28%, and 84.51% accuracy respectively over three public datasets for normal vs. NCP vs. COVID classification. This is competitive to the state-of-the-art methods. We also provide explanations in the form of GradCAM attributions, which are well aligned with expert annotations. The attributions are also seen to clearly indicate that MARL deems the peripheral regions of the lungs to be more important in the case of COVID cases while central regions are seen as more important in NCP cases. This observation matches the criteria described by radiologists in clinical literature, thereby attesting to the utility of the derived explanations.

Melanoma Classification Using a Novel Deep Convolutional Neural Network with Dermoscopic Images

Automatic melanoma detection from dermoscopic skin samples is a very challenging task. However, using a deep learning approach as a machine vision tool can overcome some challenges. This research proposes an automated melanoma classifier based on a deep convolutional neural network (DCNN) to accurately classify malignant vs. benign melanoma. The structure of the DCNN is carefully designed by organizing many layers that are responsible for extracting low to high-level features of the skin images in a unique fashion. Other vital criteria in the design of DCNN are the selection of multiple filters and their sizes, employing proper deep learning layers, choosing the depth of the network, and optimizing hyperparameters. The primary objective is to propose a lightweight and less complex DCNN than other state-of-the-art methods to classify melanoma skin cancer with high efficiency. For this study, dermoscopic images containing different cancer samples were obtained from the International Skin Imaging Collaboration datastores (ISIC 2016, ISIC2017, and ISIC 2020). We evaluated the model based on accuracy, precision, recall, specificity, and F1-score. The proposed DCNN classifier achieved accuracies of 81.41%, 88.23%, and 90.42% on the ISIC 2016, 2017, and 2020 datasets, respectively, demonstrating high performance compared with the other state-of-the-art networks. Therefore, this proposed approach could provide a less complex and advanced framework for automating the melanoma diagnostic process and expediting the identification process to save a life.

Hybrid attention network with appraiser-guided loss for counterfeit luxury handbag detection

Recently, convolutional neural networks have shown good performance in many counterfeit detection tasks. However, accurate counterfeit detection is still challenging due to the following three issues: (1) fine-grained classification, (2) class imbalance, and (3) high imitation samples. To address these issues, we propose a hybrid attention network (HANet) for counterfeit luxury handbag detection. In HANet, a hybrid attention module is first designed. Compared with existing methods that directly use classic CNNs for counterfeit detection, the HA module jointly uses a channel attention unit and a spatial attention unit to learn important information on both the channel and spatial dimensions. The HA modules can be easily integrated into the ResNet architecture to enhance the discriminative representation ability of CNNs, so as to help the network find subtle differences between the real and counterfeit products. In addition, an appraiser-guided loss is proposed to train HANet. Considering the factor of class imbalance and high imitation samples, the proposed loss gives the counterfeit class a higher weighting, and meanwhile gives the high imitation samples a much higher weighting. The proposed loss introduces the knowledge of appraisers, which allows HANet to not only treat real and counterfeit samples relatively fairly, but also pay more attention to the learning of difficult samples. To evaluate the performance of our method, we have constructed a well-benchmarked luxury handbag dataset. On this dataset, the performance of HANet, ResNet50, and the state-of-the-art attention methods is compared. The results demonstrate that HANet achieve superior performance against all its competitors.

A3 CLNN: Spatial, Spectral and Multiscale Attention ConvLSTM Neural Network for Multisource Remote Sensing Data Classification

The problem of effectively exploiting the information multiple data sources has become a relevant but challenging research topic in remote sensing. In this article, we propose a new approach to exploit the complementarity of two data sources: hyperspectral images (HSIs) and light detection and ranging (LiDAR) data. Specifically, we develop a new dual-channel spatial, spectral and multiscale attention convolutional long short-term memory neural network (called dual-channel A³ CLNN) for feature extraction and classification of multisource remote sensing data. Spatial, spectral, and multiscale attention mechanisms are first designed for HSI and LiDAR data in order to learn spectral- and spatial-enhanced feature representations and to represent multiscale information for different classes. In the designed fusion network, a novel composite attention learning mechanism (combined with a three-level fusion strategy) is used to fully integrate the features in these two data sources. Finally, inspired by the idea of transfer learning, a novel stepwise training strategy is designed to yield a final classification result. Our experimental results, conducted on several multisource remote sensing data sets, demonstrate that the newly proposed dual-channel A ³ CLNN exhibits better feature representation ability (leading to more competitive classification performance) than other state-of-the-art methods.

Multiclass Skin Lesion Classification Using Hybrid Deep Features Selection and Extreme Learning Machine

The variation in skin textures and injuries, as well as the detection and classification of skin cancer, is a difficult task. Manually detecting skin lesions from dermoscopy images is a difficult and time-consuming process. Recent advancements in the domains of the internet of things (IoT) and artificial intelligence for medical applications demonstrated improvements in both accuracy and computational time. In this paper, a new method for multiclass skin lesion classification using best deep learning feature fusion and an extreme learning machine is proposed. The proposed method includes five primary steps: image acquisition and contrast enhancement; deep learning feature extraction using transfer learning; best feature selection using hybrid whale optimization and entropy-mutual information (EMI) approach; fusion of selected features using a modified canonical correlation based approach; and, finally, extreme learning machine based classification. The feature selection step improves the system's computational efficiency and accuracy. The experiment is carried out on two publicly available datasets, HAM10000 and ISIC2018. The achieved accuracy on both datasets is 93.40 and 94.36 percent. When compared to state-of-the-art (SOTA) techniques, the proposed method's accuracy is improved. Furthermore, the proposed method is computationally efficient.

Two-Stage Deep Neural Network via Ensemble Learning for Melanoma Classification

Melanoma is a skin disease with a high fatality rate. Early diagnosis of melanoma can effectively increase the survival rate of patients. There are three types of dermoscopy images, malignant melanoma, benign nevis, and seborrheic keratosis, so using dermoscopy images to classify melanoma is an indispensable task in diagnosis. However, early melanoma classification works can only use the low-level information of images, so the melanoma cannot be classified efficiently; the recent deep learning methods mainly depend on a single network, although it can extract high-level features, the poor scale and type of the features limited the results of the classification. Therefore, we need an automatic classification method for melanoma, which can make full use of the rich and deep feature information of images for classification. In this study, we propose an ensemble method that can integrate different types of classification networks for melanoma classification. Specifically, we first use U-net to segment the lesion area of images to generate a lesion mask, thus resize images to focus on the lesion; then, we use five excellent classification models to classify dermoscopy images, and adding squeeze-excitation block (SE block) to models to emphasize the more informative features; finally, we use our proposed new ensemble network to integrate five different classification results. The experimental results prove the validity of our results. We test our method on the ISIC 2017 challenge dataset and obtain excellent results on multiple metrics; especially, we get 0.909 on accuracy. Our classification framework can provide an efficient and accurate way for melanoma classification using dermoscopy images, laying the foundation for early diagnosis and later treatment of melanoma.

Deep metric attention learning for skin lesion classification in dermoscopy images

Currently, convolutional neural networks (CNNs) have made remarkable achievements in skin lesion classification because of their end-to-end feature representation abilities. However, precise skin lesion classification is still challenging because of the following three issues: (1) insufficient training samples, (2) inter-class similarities and intra-class variations, and (3) lack of the ability to focus on discriminative skin lesion parts. To address these issues, we propose a deep metric attention learning CNN (DeMAL-CNN) for skin lesion classification. In DeMAL-CNN, a triplet-based network (TPN) is first designed based on deep metric learning, which consists of three weight-shared embedding extraction networks. TPN adopts a triplet of samples as input and uses the triplet loss to optimize the embeddings, which can not only increase the number of training samples, but also learn the embeddings robust to inter-class similarities and intra-class variations. In addition, a mixed attention mechanism considering both the spatial-wise and channel-wise attention information is designed and integrated into the construction of each embedding extraction network, which can further strengthen the skin lesion localization ability of DeMAL-CNN. After extracting the embeddings, three weight-shared classification layers are used to generate the final predictions. In the training procedure, we combine the triplet loss with the classification loss as a hybrid loss to train DeMAL-CNN. We compare DeMAL-CNN with the baseline method, attention methods, advanced challenge methods, and state-of-the-art skin lesion classification methods on the ISIC 2016 and ISIC 2017 datasets, and test its generalization ability on the PH2 dataset. The results demonstrate its effectiveness.

Feature Fusion for Diagnosis of Atypical Hepatocellular Carcinoma in Contrast- Enhanced Ultrasound

Contrast-enhanced ultrasound (CEUS) is generally employed for focal liver lesions (FLLs) diagnosis. Among the FLLs, atypical hepatocellular carcinoma (HCC) is difficult to distinguish from focal nodular hyperplasia (FNH) in CEUS video. For this reason, we propose and evaluate a feature fusion method to resolve this problem. The proposed algorithm extracts a set of hand-crafted features and the deep features from the CEUS cine clip data. The hand-crafted features include the spatial-temporal feature based on a novel descriptor called Velocity-Similarity and Dissimilarity Matching Local Binary Pattern (V-SDMLBP), and the deep features from a 3-D convolution neural network (3D-CNN). Then the two types of features are fused. Finally, a classifier is employed to diagnose HCC or FNH. Several classifiers have achieved excellent performance, which demonstrates the superiority of the fused features. In addition, compared with general CNNs, the proposed fused features have better interpretability.

Efficient structural pseudoinverse learning-based hierarchical representation learning for skin lesion classification

The success of deep learning in skin lesion classification mainly depends on the ultra-deep neural network and the significantly large training data set. Deep learning training is usually time-consuming, and large datasets with labels are hard to obtain, especially skin lesion images. Although pre-training and data augmentation can alleviate these issues, there are still some problems: (1) the data domain is not consistent, resulting in the slow convergence; and (2) low robustness to confusing skin lesions. To solve these problems, we propose an efficient structural pseudoinverse learning-based hierarchical representation learning method. Preliminary feature extraction, shallow network feature extraction and deep learning feature extraction are carried out respectively before the classification of skin lesion images. Gabor filter and pre-trained deep convolutional neural network are used for preliminary feature extraction. The structural pseudoinverse learning (S-PIL) algorithm is used to extract the shallow features. Then, S-PIL preliminarily identifies the skin lesion images that are difficult to be classified to form a new training set for deep learning feature extraction. Through the hierarchical representation learning, we analyze the features of skin lesion images layer by layer to improve the final classification. Our method not only avoid the slow convergence caused by inconsistency of data domain but also enhances the training of confusing examples. Without using additional data, our approach outperforms existing methods in the ISIC 2017 and ISIC 2018 datasets.

Characterizing Malignant Melanoma Clinically Resembling Seborrheic Keratosis Using Deep Knowledge Transfer

Simple Summary

Malignant melanomas (MMs) with aypical clinical presentation constitute a diagnostic pitfall, and false negatives carry the risk of a diagnostic delay and improper disease management. Among the most common, challenging presentation forms of MMs are those that clinically resemble seborrheic keratosis (SK). On the other hand, SK may mimic melanoma, producing ‘false positive overdiagnosis’ and leading to needless excisions. The evolving efficiency of deep learning algorithms in image recognition and the availability of large image databases have accelerated the development of advanced computer-aided systems for melanoma detection. In the present study, we used image data from the International Skin Image Collaboration archive to explore the capacity of deep knowledge transfer in the challenging diagnostic task of the atypical skin tumors of MM and SK.

Abstract

Malignant melanomas resembling seborrheic keratosis (SK-like MMs) are atypical, challenging to diagnose melanoma cases that carry the risk of delayed diagnosis and inadequate treatment. On the other hand, SK may mimic melanoma, producing a ‘false positive’ with unnecessary lesion excisions. The present study proposes a computer-based approach using dermoscopy images for the characterization of SΚ-like MMs. Dermoscopic images were retrieved from the International Skin Imaging Collaboration archive. Exploiting image embeddings from pretrained convolutional network VGG16, we trained a support vector machine (SVM) classification model on a data set of 667 images. SVM optimal hyperparameter selection was carried out using the Bayesian optimization method. The classifier was tested on an independent data set of 311 images with atypical appearance: MMs had an absence of pigmented network and had an existence of milia-like cysts. SK lacked milia-like cysts and had a pigmented network. Atypical MMs were characterized with a sensitivity and specificity of 78.6% and 84.5%, respectively. The advent of deep learning in image recognition has attracted the interest of computer science towards improved skin lesion diagnosis. Open-source, public access archives of skin images empower further the implementation and validation of computer-based systems that might contribute significantly to complex clinical diagnostic problems such as the characterization of SK-like MMs.

Artificial Intelligence for Skin Cancer Detection: Scoping Review

BACKGROUND

Skin cancer is the most common cancer type affecting humans. Traditional skin cancer diagnosis methods are costly, require a professional physician, and take time. Hence, to aid in diagnosing skin cancer, artificial intelligence (AI) tools are being used, including shallow and deep machine learning-based methodologies that are trained to detect and classify skin cancer using computer algorithms and deep neural networks.

OBJECTIVE

The aim of this study was to identify and group the different types of AI-based technologies used to detect and classify skin cancer. The study also examined the reliability of the selected papers by studying the correlation between the data set size and the number of diagnostic classes with the performance metrics used to evaluate the models.

METHODS

We conducted a systematic search for papers using Institute of Electrical and Electronics Engineers (IEEE) Xplore, Association for Computing Machinery Digital Library (ACM DL), and Ovid MEDLINE databases following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines. The studies included in this scoping review had to fulfill several selection criteria: being specifically about skin cancer, detecting or classifying skin cancer, and using AI technologies. Study selection and data extraction were independently conducted by two reviewers. Extracted data were narratively synthesized, where studies were grouped based on the diagnostic AI techniques and their evaluation metrics.

RESULTS

We retrieved 906 papers from the 3 databases, of which 53 were eligible for this review. Shallow AI-based techniques were used in 14 studies, and deep AI-based techniques were used in 39 studies. The studies used up to 11 evaluation metrics to assess the proposed models, where 39 studies used accuracy as the primary evaluation metric. Overall, studies that used smaller data sets reported higher accuracy.

CONCLUSIONS

This paper examined multiple AI-based skin cancer detection models. However, a direct comparison between methods was hindered by the varied use of different evaluation metrics and image types. Performance scores were affected by factors such as data set size, number of diagnostic classes, and techniques. Hence, the reliability of shallow and deep models with higher accuracy scores was questionable since they were trained and tested on relatively small data sets of a few diagnostic classes.

FusionM4Net: A multi-stage multi-modal learning algorithm for multi-label skin lesion classification

Skin disease is one of the most common diseases in the world. Deep learning-based methods have achieved excellent skin lesion recognition performance, most of which are based on only dermoscopy images. In recent works that use multi-modality data (patient's meta-data, clinical images, and dermoscopy images), the methods adopt a one-stage fusion approach and only optimize the information fusion at the feature level. These methods do not use information fusion at the decision level and thus cannot fully use the data of all modalities. This work proposes a novel two-stage multi-modal learning algorithm (FusionM4Net) for multi-label skin diseases classification. At the first stage, we construct a FusionNet, which exploits and integrates the representation of clinical and dermoscopy images at the feature level, and then uses a Fusion Scheme 1 to conduct the information fusion at the decision level. At the second stage, to further incorporate the patient's meta-data, we propose a Fusion Scheme 2, which integrates the multi-label predictive information from the first stage and patient's meta-data information to train an SVM cluster. The final diagnosis is formed by the fusion of the predictions from the first and second stages. Our algorithm was evaluated on the seven-point checklist dataset, a well-established multi-modality multi-label skin disease dataset. Without using the patient's meta-data, the proposed FusionM4Net's first stage (FusionM4Net-FS) achieved an average accuracy of 75.7% for multi-classification tasks and 74.9% for diagnostic tasks, which is more accurate than other state-of-the-art methods. By further fusing the patient's meta-data at FusionM4Net's second stage (FusionM4Net-SS), the entire FusionM4Net finally boosts the average accuracy to 77.0% and the diagnostic accuracy to 78.5%, which indicates its robust and excellent classification performance on the label-imbalanced dataset. The corresponding code is available at: https://github.com/pixixiaonaogou/MLSDR.

Improving Skin Cancer Classification Using Heavy-Tailed Student T-Distribution in Generative Adversarial Networks (TED-GAN)

Deep learning has gained immense attention from researchers in medicine, especially in medical imaging. The main bottleneck is the unavailability of sufficiently large medical datasets required for the good performance of deep learning models. This paper proposes a new framework consisting of one variational autoencoder (VAE), two generative adversarial networks, and one auxiliary classifier to artificially generate realistic-looking skin lesion images and improve classification performance. We first train the encoder-decoder network to obtain the latent noise vector with the image manifold’s information and let the generative adversarial network sample the input from this informative noise vector in order to generate the skin lesion images. The use of informative noise allows the GAN to avoid mode collapse and creates faster convergence. To improve the diversity in the generated images, we use another GAN with an auxiliary classifier, which samples the noise vector from a heavy-tailed student t-distribution instead of a random noise Gaussian distribution. The proposed framework was named TED-GAN, with T from the t-distribution and ED from the encoder-decoder network which is part of the solution. The proposed framework could be used in a broad range of areas in medical imaging. We used it here to generate skin lesion images and have obtained an improved classification performance on the skin lesion classification task, rising from 66% average accuracy to 92.5%. The results show that TED-GAN has a better impact on the classification task because of its diverse range of generated images due to the use of a heavy-tailed t-distribution.

Multi-level Attentive Skin Lesion Learning for Melanoma Classification

Melanoma classification plays an important role in skin lesion diagnosis. Nevertheless, melanoma classification is a challenging task, due to the appearance variation of the skin lesions, and the interference of the noises from dermoscopic imaging. In this paper, we propose a multi-level attentive skin lesion learning (MASLL) network to enhance melanoma classification. Specifically, we design a local learning branch with a skin lesion localization (SLL) module to assist the network in learning the lesion features from the region of interest. In addition, we propose a weighted feature integration (WFI) module to fuse the lesion information from the global and local branches, which further enhances the feature discriminative capability of the skin lesions. Experimental results on ISIC 2017 dataset show the effectiveness of the proposed method on melanoma classification.

SCOAT-Net: A novel network for segmenting COVID-19 lung opacification from CT images

Automatic segmentation of lung opacification from computed tomography (CT) images shows excellent potential for quickly and accurately quantifying the infection of Coronavirus disease 2019 (COVID-19) and judging the disease development and treatment response. However, some challenges still exist, including the complexity and variability features of the opacity regions, the small difference between the infected and healthy tissues, and the noise of CT images. Due to limited medical resources, it is impractical to obtain a large amount of data in a short time, which further hinders the training of deep learning models. To answer these challenges, we proposed a novel spatial- and channel-wise coarse-to-fine attention network (SCOAT-Net), inspired by the biological vision mechanism, for the segmentation of COVID-19 lung opacification from CT images. With the UNet++ as basic structure, our SCOAT-Net introduces the specially designed spatial-wise and channel-wise attention modules, which serve to collaboratively boost the attention learning of the network and extract the efficient features of the infected opacification regions at the pixel and channel levels. Experiments show that our proposed SCOAT-Net achieves better results compared to several state-of-the-art image segmentation networks and has acceptable generalization ability.

Deep attention branch networks for skin lesion classification

BACKGROUND AND OBJECTIVE

The skin lesion usually covers a small region of the dermoscopy image, and the lesions of different categories might own high similarities. Therefore, it is essential to design an elaborate network for accurate skin lesion classification, which can focus on semantically meaningful lesion parts. Although the Class Activation Mapping (CAM) shows good localization capability of highlighting the discriminative parts, it cannot be obtained in the forward propagation process.

METHODS

We propose a Deep Attention Branch Network (DABN) model, which introduces the attention branches to expand the conventional Deep Convolutional Neural Networks (DCNN). The attention branch is designed to obtain the CAM in the training stage, which is then utilized as an attention map to make the network focus on discriminative parts of skin lesions. DABN is applicable to multiple DCNN structures and can be trained in an end-to-end manner. Moreover, a novel Entropy-guided Loss Weighting (ELW) strategy is designed to counter class imbalance influence in the skin lesion datasets.

RESULTS

The proposed method achieves an Average Precision (AP) of 0.719 on the ISIC-2016 dataset and an average area under the ROC curve (AUC) of 0.922 on the ISIC-2017 dataset. Compared with other state-of-the-art methods, our method obtains better performance without external data and ensemble learning. Moreover, extensive experiments demonstrate that it can be applied to multi-class classification tasks and improves mean sensitivity by more than 2.6% in different DCNN structures.

CONCLUSIONS

The proposed method can adaptively focus on the discriminative regions of dermoscopy images and allows for effective training when facing class imbalance, leading to the performance improvement of skin lesion classification, which could also be applied to other clinical applications.

IMAL-Net: Interpretable multi-task attention learning network for invasive lung adenocarcinoma screening in CT images

PURPOSE

Feature maps created from deep convolutional neural networks (DCNNs) have been widely used for visual explanation of DCNN-based classification tasks. However, many clinical applications such as benign-malignant classification of lung nodules normally require quantitative and objective interpretability, rather than just visualization. In this paper, we propose a novel interpretable multi-task attention learning network named IMAL-Net for early invasive adenocarcinoma screening in chest computed tomography images, which takes advantage of segmentation prior to assist interpretable classification.

METHODS

Two sub-ResNets are firstly integrated together via a prior-attention mechanism for simultaneous nodule segmentation and invasiveness classification. Then, numerous radiomic features from the segmentation results are concatenated with high-level semantic features from the classification subnetwork by FC layers to achieve superior performance. Meanwhile, an end-to-end feature selection mechanism (named FSM) is designed to quantify crucial radiomic features greatly affecting the prediction of each sample, and thus it can provide clinically applicable interpretability to the prediction result.

RESULTS

Nodule samples from a total of 1626 patients were collected from two grade-A hospitals for large-scale verification. Five-fold cross validation demonstrated that the proposed IMAL-Net can achieve an AUC score of 93.8% ± 1.1% and a recall score of 93.8% ± 2.8% for identification of invasive lung adenocarcinoma.

CONCLUSIONS

It can be concluded that fusing semantic features and radiomic features can achieve obvious improvements in the invasiveness classification task. Moreover, by learning more fine-grained semantic features and highlighting the most important radiomics features, the proposed attention and FSM mechanisms not only can further improve the performance but also can be used for both visual explanations and objective analysis of the classification results.

Skin Lesion Detection Algorithms in Whole Body Images

Melanoma is one of the most lethal and rapidly growing cancers, causing many deaths each year. This cancer can be treated effectively if it is detected quickly. For this reason, many algorithms and systems have been developed to support automatic or semiautomatic detection of neoplastic skin lesions based on the analysis of optical images of individual moles. Recently, full-body systems have gained attention because they enable the analysis of the patient’s entire body based on a set of photos. This paper presents a prototype of such a system, focusing mainly on assessing the effectiveness of algorithms developed for the detection and segmentation of lesions. Three detection algorithms (and their fusion) were analyzed, one implementing deep learning methods and two classic approaches, using local brightness distribution and a correlation method. For fusion of algorithms, detection sensitivity = 0.95 and precision = 0.94 were obtained. Moreover, the values of the selected geometric parameters of segmented lesions were calculated and compared for all algorithms. The obtained results showed a high accuracy of the evaluated parameters (error of area estimation <10%), especially for lesions with dimensions greater than 3 mm, which are the most suspected of being neoplastic lesions.

Densely connected attention network for diagnosing COVID-19 based on chest CT

BACKGROUND

To fully enhance the feature extraction capabilities of deep learning models, so as to accurately diagnose coronavirus disease 2019 (COVID-19) based on chest CT images, a densely connected attention network (DenseANet) was constructed by utilizing the self-attention mechanism in deep learning.

METHODS

During the construction of the DenseANet, we not only densely connected attention features within and between the feature extraction blocks with the same scale, but also densely connected attention features with different scales at the end of the deep model, thereby further enhancing the high-order features. In this way, as the depth of the deep model increases, the spatial attention features generated by different layers can be densely connected and gradually transferred to deeper layers. The DenseANet takes CT images of the lung fields segmented by an improved U-Net as inputs and outputs the probability of the patients suffering from COVID-19.

RESULTS

Compared with exiting attention networks, DenseANet can maximize the utilization of self-attention features at different depths in the model. A five-fold cross-validation experiment was performed on a dataset containing 2993 CT scans of 2121 patients, and experiments showed that the DenseANet can effectively locate the lung lesions of patients infected with SARS-CoV-2, and distinguish COVID-19, common pneumonia and normal controls with an average of 96.06% Acc and 0.989 AUC.

CONCLUSIONS

The DenseANet we proposed can generate strong attention features and achieve the best diagnosis results. In addition, the proposed method of densely connecting attention features can be easily extended to other advanced deep learning methods to improve their performance in related tasks.

Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation

Multimodal positron emission tomography-computed tomography (PET-CT) is used routinely in the assessment of cancer. PET-CT combines the high sensitivity for tumor detection of PET and anatomical information from CT. Tumor segmentation is a critical element of PET-CT but at present, the performance of existing automated methods for this challenging task is low. Segmentation tends to be done manually by different imaging experts, which is labor-intensive and prone to errors and inconsistency. Previous automated segmentation methods largely focused on fusing information that is extracted separately from the PET and CT modalities, with the underlying assumption that each modality contains complementary information. However, these methods do not fully exploit the high PET tumor sensitivity that can guide the segmentation. We introduce a deep learning-based framework in multimodal PET-CT segmentation with a multimodal spatial attention module (MSAM). The MSAM automatically learns to emphasize regions (spatial areas) related to tumors and suppress normal regions with physiologic high-uptake from the PET input. The resulting spatial attention maps are subsequently employed to target a convolutional neural network (CNN) backbone for segmentation of areas with higher tumor likelihood from the CT image. Our experimental results on two clinical PET-CT datasets of non-small cell lung cancer (NSCLC) and soft tissue sarcoma (STS) validate the effectiveness of our framework in these different cancer types. We show that our MSAM, with a conventional U-Net backbone, surpasses the state-of-the-art lung tumor segmentation approach by a margin of 7.6% in Dice similarity coefficient (DSC).

Two-Stream Compare and Contrast Network for Vertebral Compression Fracture Diagnosis

Differentiating Vertebral Compression Fractures (VCFs) associated with trauma and osteoporosis (benign VCFs) or those caused by metastatic cancer (malignant VCFs) is critically important for treatment decisions. So far, automatic VCFs diagnosis is solved in a two-step manner, i.e., first identify VCFs and then classify them into benign or malignant. In this paper, we explore to model VCFs diagnosis as a three-class classification problem, i.e., normal vertebrae, benign VCFs, and malignant VCFs. However, VCFs recognition and classification require very different features, and both tasks are characterized by high intra-class variation and high inter-class similarity. Moreover, the dataset is extremely class-imbalanced. To address the above challenges, we propose a novel Two-Stream Compare and Contrast Network (TSCCN) for VCFs diagnosis. This network consists of two streams, a recognition stream which learns to identify VCFs through comparing and contrasting between adjacent vertebrae, and a classification stream which compares and contrasts between intra-class and inter-class to learn features for fine-grained classification. The two streams are integrated via a learnable weight control module which adaptively sets their contribution. TSCCN is evaluated on a dataset consisting of 239 VCFs patients and achieves the average sensitivity and specificity of 92.56% and 96.29%, respectively.

Machine Learning and Deep Learning Methods for Skin Lesion Classification and Diagnosis: A Systematic Review

Computer-aided systems for skin lesion diagnosis is a growing area of research. Recently, researchers have shown an increasing interest in developing computer-aided diagnosis systems. This paper aims to review, synthesize and evaluate the quality of evidence for the diagnostic accuracy of computer-aided systems. This study discusses the papers published in the last five years in ScienceDirect, IEEE, and SpringerLink databases. It includes 53 articles using traditional machine learning methods and 49 articles using deep learning methods. The studies are compared based on their contributions, the methods used and the achieved results. The work identified the main challenges of evaluating skin lesion segmentation and classification methods such as small datasets, ad hoc image selection and racial bias.

Using deep learning for dermatologist-level detection of suspicious pigmented skin lesions from wide-field images

A reported 96,480 people were diagnosed with melanoma in the United States in 2019, leading to 7230 reported deaths. Early-stage identification of suspicious pigmented lesions (SPLs) in primary care settings can lead to improved melanoma prognosis and a possible 20-fold reduction in treatment cost. Despite this clinical and economic value, efficient tools for SPL detection are mostly absent. To bridge this gap, we developed an SPL analysis system for wide-field images using deep convolutional neural networks (DCNNs) and applied it to a 38,283 dermatological dataset collected from 133 patients and publicly available images. These images were obtained from a variety of consumer-grade cameras (15,244 nondermoscopy) and classified by three board-certified dermatologists. Our system achieved more than 90.3% sensitivity (95% confidence interval, 90 to 90.6) and 89.9% specificity (89.6 to 90.2%) in distinguishing SPLs from nonsuspicious lesions, skin, and complex backgrounds, avoiding the need for cumbersome individual lesion imaging. We also present a new method to extract intrapatient lesion saliency (ugly duckling criteria) on the basis of DCNN features from detected lesions. This saliency ranking was validated against three board-certified dermatologists using a set of 135 individual wide-field images from 68 dermatological patients not included in the DCNN training set, exhibiting 82.96% (67.88 to 88.26%) agreement with at least one of the top three lesions in the dermatological consensus ranking. This method could allow for rapid and accurate assessments of pigmented lesion suspiciousness within a primary care visit and could enable improved patient triaging, utilization of resources, and earlier treatment of melanoma.

Harmony Loss for Unbalanced Prediction

In medical image analysis, in order to reduce the impact of unbalanced data sets on data-driven deep learning models, according to the characteristic that the area under the Precision-Recall curve (AUC_PR) is sensitive to each category of samples, a novel Harmony loss function with fast convergence speed and high stability was constructed. Since AUC_PR needs to be calculated in discrete domain, in order to ensure the continuous differentiability and gradient existence of the Harmony loss, first, the Logistic function was used to approximate the Logical function in AUC_PR. Then, to improve the optimization speed of the Harmony loss during model training, a method of manually setting a certain number of classification thresholds was proposed to further approximate the calculation of AUC_PR. After the above two approximate calculation processes, the Harmony loss with stable gradient and high computational efficiency was designed. In the optimization process of the model, since Harmony loss can reconcile recall and precision of each category under different classification thresholds, thereby, it can not only improve the models ability to recognize categories with less samples, but also maintain the stability of the training curve. To comprehensively evaluate the effects of Harmony loss function, we performed experiments on image 3D reconstruction, 2D segmentation, and unbalanced classification tasks. Experimental results showed that the Harmony loss achieved the state-of-the-art results on four unbalanced data sets. Moreover, the Harmony loss can be easily combined with existing loss functions, and is suitable for most common deep learning models.

Artificial intelligence in tumor subregion analysis based on medical imaging: A review

Medical imaging is widely used in the diagnosis and treatment of cancer, and artificial intelligence (AI) has achieved tremendous success in medical image analysis. This paper reviews AI-based tumor subregion analysis in medical imaging. We summarize the latest AI-based methods for tumor subregion analysis and their applications. Specifically, we categorize the AI-based methods by training strategy: supervised and unsupervised. A detailed review of each category is presented, highlighting important contributions and achievements. Specific challenges and potential applications of AI in tumor subregion analysis are discussed.

Multiscale Attention Guided Network for COVID-19 Diagnosis Using Chest X-Ray Images

Coronavirus disease 2019 (COVID-19) is one of the most destructive pandemic after millennium, forcing the world to tackle a health crisis. Automated lung infections classification using chest X-ray (CXR) images could strengthen diagnostic capability when handling COVID-19. However, classifying COVID-19 from pneumonia cases using CXR image is a difficult task because of shared spatial characteristics, high feature variation and contrast diversity between cases. Moreover, massive data collection is impractical for a newly emerged disease, which limited the performance of data thirsty deep learning models. To address these challenges, Multiscale Attention Guided deep network with Soft Distance regularization (MAG-SD) is proposed to automatically classify COVID-19 from pneumonia CXR images. In MAG-SD, MA-Net is used to produce prediction vector and attention from multiscale feature maps. To improve the robustness of trained model and relieve the shortage of training data, attention guided augmentations along with a soft distance regularization are posed, which aims at generating meaningful augmentations and reduce noise. Our multiscale attention model achieves better classification performance on our pneumonia CXR image dataset. Plentiful experiments are proposed for MAG-SD which demonstrates its unique advantage in pneumonia classification over cutting-edge models. The code is available at https://github.com/JasonLeeGHub/MAG-SD.

Deep Learning Classifier with Patient's Metadata of Dermoscopic Images in Malignant Melanoma Detection

BACKGROUND

Incidence of skin cancer is one of the global burdens of malignancies that increase each year, with melanoma being the deadliest one. Imaging-based automated skin cancer detection still remains challenging owing to variability in the skin lesions and limited standard dataset availability. Recent research indicates the potential of deep convolutional neural networks (CNN) in predicting outcomes from simple as well as highly complicated images. However, its implementation requires high-class computational facility, that is not feasible in low resource and remote areas of health care. There is potential in combining image and patient's metadata, but the study is still lacking.

OBJECTIVE

We want to develop malignant melanoma detection based on dermoscopic images and patient's metadata using an artificial intelligence (AI) model that will work on low-resource devices.

METHODS

We used an open-access dermatology repository of International Skin Imaging Collaboration (ISIC) Archive dataset consist of 23,801 biopsy-proven dermoscopic images. We tested performance for binary classification malignant melanomas vs nonmalignant melanomas. From 1200 sample images, we split the data for training (72%), validation (18%), and testing (10%). We compared CNN with image data only (CNN model) vs CNN for image data combined with an artificial neural network (ANN) for patient's metadata (CNN+ANN model).

RESULTS

The balanced accuracy for CNN+ANN model was higher (92.34%) than the CNN model (73.69%). Combination of the patient's metadata using ANN prevents the overfitting that occurs in the CNN model using dermoscopic images only. This small size (24 MB) of this model made it possible to run on a medium class computer without the need of cloud computing, suitable for deployment on devices with limited resources.

CONCLUSION

The CNN+ANN model can increase the accuracy of classification in malignant melanoma detection even with limited data and is promising for development as a screening device in remote and low resources health care.

Attention-Aware Residual Network Based Manifold Learning for White Blood Cells Classification

The classification of six types of white blood cells (WBCs) is considered essential for leukemia diagnosis, while the classification is labor-intensive and strict with the clinical experience. To relieve the complicated process with an efficient and automatic method, we propose the Attention-aware Residual Network based Manifold Learning model (ARML) to classify WBCs. The proposed ARML model leverages the adaptive attention-aware residual learning to exploit the category-relevant image-level features and strengthen the first-order feature representation ability. To learn more discriminatory information than the first-order ones, the second-order features are characterized. Afterwards, ARML encodes both the first- and second-order features with Gaussian embedding into the Riemannian manifold to learn the underlying non-linear structure of the features for classification. ARML can be trained in an end-to-end fashion, and the learnable parameters are iteratively optimized. 10800 WBCs images (1800 images for each type) is collected, 9000 images and five-fold cross-validation are used for training and validation of the model, while additional 1800 images for testing. The results show that ARML achieving average classification accuracy of 0.953 outperforms other state-of-the-art methods with fewer trainable parameters. In the ablation study, ARML achieves improved accuracy against its three variants: without manifold learning (AR), without attention-aware learning (RML), and AR without attention-aware learning. The t-SNE results illustrate that ARML has learned more distinguishable features than the comparison methods, which benefits the WBCs classification. ARML provides a clinically feasible WBCs classification solution for leukemia diagnose with an efficient manner.

Cross-attention multi-branch network for fundus diseases classification using SLO images

Fundus diseases classification is vital for the health of human beings. However, most of existing methods detect diseases by means of single angle fundus images, which lead to the lack of pathological information. To address this limitation, this paper proposes a novel deep learning method to complete different fundus diseases classification tasks using ultra-wide field scanning laser ophthalmoscopy (SLO) images, which have an ultra-wide field view of 180-200˚. The proposed deep model consists of multi-branch network, atrous spatial pyramid pooling module (ASPP), cross-attention and depth-wise attention module. Specifically, the multi-branch network employs the ResNet-34 model as the backbone to extract feature information, where the ResNet-34 model with two-branch is followed by the ASPP module to extract multi-scale spatial contextual features by setting different dilated rates. The depth-wise attention module can provide the global attention map from the multi-branch network, which enables the network to focus on the salient targets of interest. The cross-attention module adopts the cross-fusion mode to fuse the channel and spatial attention maps from the ResNet-34 model with two-branch, which can enhance the representation ability of the disease-specific features. The extensive experiments on our collected SLO images and two publicly available datasets demonstrate that the proposed method can outperform the state-of-the-art methods and achieve quite promising classification performance of the fundus diseases.

Short-Term Lesion Change Detection for Melanoma Screening With Novel Siamese Neural Network

Short-term monitoring of lesion changes has been a widely accepted clinical guideline for melanoma screening. When there is a significant change of a melanocytic lesion at three months, the lesion will be excised to exclude melanoma. However, the decision on change or no-change heavily depends on the experience and bias of individual clinicians, which is subjective. For the first time, a novel deep learning based method is developed in this paper for automatically detecting short-term lesion changes in melanoma screening. The lesion change detection is formulated as a task measuring the similarity between two dermoscopy images taken for a lesion in a short time-frame, and a novel Siamese structure based deep network is proposed to produce the decision: changed (i.e. not similar) or unchanged (i.e. similar enough). Under the Siamese framework, a novel structure, namely Tensorial Regression Process, is proposed to extract the global features of lesion images, in addition to deep convolutional features. In order to mimic the decision-making process of clinicians who often focus more on regions with specific patterns when comparing a pair of lesion images, a segmentation loss (SegLoss) is further devised and incorporated into the proposed network as a regularization term. To evaluate the proposed method, an in-house dataset with 1,000 pairs of lesion images taken in a short time-frame at a clinical melanoma centre was established. Experimental results on this first-of-a-kind large dataset indicate that the proposed model is promising in detecting the short-term lesion change for objective melanoma screening.