A multi-stage multi-modal learning algorithm with adaptive multimodal fusion for improving multi-label skin lesion classification

Skin cancer is frequently occurring and has become a major contributor to both cancer incidence and mortality. Accurate and timely diagnosis of skin cancer holds the potential to save lives. Deep learning-based methods have demonstrated significant advancements in the screening of skin cancers. However, most current approaches rely on a single modality input for diagnosis, thereby missing out on valuable complementary information that could enhance accuracy. Although some multimodal-based methods exist, they often lack adaptability and fail to fully leverage multimodal information. In this paper, we introduce a novel uncertainty-based hybrid fusion strategy for a multi-modal learning algorithm aimed at skin cancer diagnosis. Our approach specifically combines three different modalities: clinical images, dermoscopy images, and metadata, to make the final classification. For the fusion of two image modalities, we employ an intermediate fusion strategy that considers the similarity between clinical and dermoscopy images to extract features containing both complementary and correlated information. To capture the correlated information, we utilize cosine similarity, and we employ concatenation as the means for integrating complementary information. In the fusion of image and metadata modalities, we leverage uncertainty to obtain confident late fusion results, allowing our method to adaptively combine the information from different modalities. We conducted comprehensive experiments using a popular publicly available skin disease diagnosis dataset, and the results of these experiments demonstrate the effectiveness of our proposed method. Our proposed fusion algorithm could enhance the clinical applicability of automated skin lesion classification, offering a more robust and adaptive way to make automatic diagnoses with the help of uncertainty mechanism. Code is available at https://github.com/Zuo-Lihan/CosCatNet-Adaptive_Fusion_Algorithm.

Skin Cancer Detection Using Transfer Learning and Deep Attention Mechanisms

Background/Objectives: Early and accurate diagnosis of skin cancer improves survival rates; however, dermatologists often struggle with lesion detection due to similar pigmentation. Deep learning and transfer learning models have shown promise in diagnosing skin cancers through image processing. Integrating attention mechanisms (AMs) with deep learning has further enhanced the accuracy of medical image classification. While significant progress has been made, further research is needed to improve the detection accuracy. Previous studies have not explored the integration of attention mechanisms with the pre-trained Xception transfer learning model for binary classification of skin cancer. This study aims to investigate the impact of various attention mechanisms on the Xception model's performance in detecting benign and malignant skin lesions. Methods: We conducted four experiments on the HAM10000 dataset. Three models integrated self-attention (SL), hard attention (HD), and soft attention (SF) mechanisms, while the fourth model used the standard Xception without attention mechanisms. Each mechanism analyzed features from the Xception model uniquely: self-attention examined the input relationships, hard-attention selected elements sparsely, and soft-attention distributed the focus probabilistically. Results: Integrating AMs into the Xception architecture effectively enhanced its performance. The accuracy of the Xception alone was 91.05%. With AMs, the accuracy increased to 94.11% using self-attention, 93.29% with soft attention, and 92.97% with hard attention. Moreover, the proposed models outperformed previous studies in terms of the recall metrics, which are crucial for medical investigations. Conclusions: These findings suggest that AMs can enhance performance in relation to complex medical imaging tasks, potentially supporting earlier diagnosis and improving treatment outcomes.

Artificial intelligence-based pathological application to predict regional lymph node metastasis in Papillary Thyroid Cancer

In this study, a model for predicting lymph node metastasis in papillary thyroid cancer was trained using pathology images from the TCGA(The Cancer Genome Atlas) public dataset of papillary thyroid cancer, and a front-end inference model was trained using our center's dataset based on the concept of probabilistic propagation of nodes in graph neural networks. Effectively predicting whether a tumor will spread to regional lymph nodes using a single pathological image is the capacity of the model described above. This study demonstrates that regional lymph nodes in papillary thyroid cancer are a common and predictable occurrence, providing valuable ideas for future research. Now we publish the above research process and code for further study by other researchers, and we also make the above inference algorithm public at the URL: http:// thyroid-diseases-research.com/, with the hope that other researchers will validate it and provide us with ideas or datasets for further study.

GU-Net: Causal relationship-based generative medical image segmentation model

Due to significant anatomical variations in medical images across different cases, medical image segmentation is a highly challenging task. Convolutional neural networks have shown faster and more accurate performance in medical image segmentation. However, existing networks for medical image segmentation mostly rely on independent training of the model using data samples and loss functions, lacking interactive training and feedback mechanisms. This leads to a relatively singular training approach for the models, and furthermore, some networks can only perform segmentation for specific diseases. In this paper, we propose a causal relationship-based generative medical image segmentation model named GU-Net. We integrate a counterfactual attention mechanism combined with CBAM into the decoder of U-Net as a generative network, and then combine it with a GAN network where the discriminator is used for backpropagation. This enables alternate optimization and training between the generative network and discriminator, enhancing the expressive and learning capabilities of the network model to output prediction segmentation results closer to the ground truth. Additionally, the interaction and transmission of information help the network model capture richer feature representations, extract more accurate features, reduce overfitting, and improve model stability and robustness through feedback mechanisms. Experimental results demonstrate that our proposed GU-Net network achieves better segmentation performance not only in cases with abundant data samples and relatively simple segmentation targets or high contrast between the target and background regions but also in scenarios with limited data samples and challenging segmentation tasks. Comparing with existing U-Net networks with attention mechanisms, GU-Net consistently improves Dice scores by 1.19%, 2.93%, 5.01%, and 5.50% on ISIC 2016, ISIC 2017, ISIC 2018, and Gland Segmentation datasets, respectively.

DermSynth3D: Synthesis of in-the-wild annotated dermatology images

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

STCS-Net: a medical image segmentation network that fully utilizes multi-scale information

In recent years, significant progress has been made in the field of medical image segmentation through the application of deep learning and neural networks. Numerous studies have focused on optimizing encoders to extract more comprehensive key information. However, the importance of decoders in directly influencing the final output of images cannot be overstated. The ability of decoders to effectively leverage diverse information and further refine crucial details is of paramount importance. This paper proposes a medical image segmentation architecture named STCS-Net. The designed decoder in STCS-Net facilitates multi-scale filtering and correction of information from the encoder, thereby enhancing the accuracy of extracting vital features. Additionally, an information enhancement module is introduced in skip connections to highlight essential features and improve the inter-layer information interaction capabilities. Comprehensive evaluations on the ISIC2016, ISIC2018, and Lung datasets validate the superiority of STCS-Net across different scenarios. Experimental results demonstrate the outstanding performance of STCS-Net on all three datasets. Comparative experiments highlight the advantages of our proposed network in terms of accuracy and parameter efficiency. Ablation studies confirm the effectiveness of the introduced decoder and skip connection module. This research introduces a novel approach to the field of medical image segmentation, providing new perspectives and solutions for future developments in medical image processing and analysis.

A generic plug & play diffusion-based denosing module for medical image segmentation

Medical image segmentation faces challenges because of the small sample size of the dataset and the fact that images often have noise and artifacts. In recent years, diffusion models have proven very effective in image generation and have been used widely in computer vision. This paper presents a new feature map denoising module (FMD) based on the diffusion model for feature refinement, which is plug-and-play, allowing flexible integration into popular used segmentation networks for seamless end-to-end training. We evaluate the performance of the FMD module on four models, UNet, UNeXt, TransUNet, and IB-TransUNet, by conducting experiments on four datasets. The experimental data analysis shows that adding the FMD module significantly positively impacts the model performance. Furthermore, especially for small lesion areas and minor organs, adding the FMD module allows users to obtain more accurate segmentation results than the original model.

LGI Net: Enhancing local-global information interaction for medical image segmentation

Medical image segmentation is a critical task used to accurately extract regions of interest and pathological areas from medical images. In recent years, significant progress has been made in the field of medical image segmentation using deep learning and neural networks. However, existing methods still have limitations in terms of fusing local features and global contextual information due to the complex variations and irregular shapes of medical images. To address this issue, this paper proposes a medical image segmentation architecture called LGI Net, which improves the internal computation to achieve sufficient interaction between local perceptual capabilities and global contextual information within the network. Furthermore, the network incorporates an ECA module to effectively capture the interplay between channels and improve inter-layer information exchange capabilities. We conducted extensive experiments on three public medical image datasets: Kvasir, ISIC, and X-ray to validate the effectiveness of the proposed method. Ablation studies demonstrated the effectiveness of our LGAF, and comparative experiments confirmed the superiority of our proposed LGI Net in terms of accuracy and parameter efficiency. This study provides an innovative approach in the field of medical image segmentation, offering valuable insights for further improvements in accuracy and performance. The code and models will be available at https://github.com/LiuLinjie0310/LGI-Net.

M-CSAFN: Multi-Color Space Adaptive Fusion Network for Automated Port-Wine Stains Segmentation

Automatic segmentation of port-wine stains (PWS) from clinical images is critical for accurate diagnosis and objective assessment of PWS. However, this is a challenging task due to the color heterogeneity, low contrast, and indistinguishable appearance of PWS lesions. To address such challenges, we propose a novel multi-color space adaptive fusion network (M-CSAFN) for PWS segmentation. First, a multi-branch detection model is constructed based on six typical color spaces, which utilizes rich color texture information to highlight the difference between lesions and surrounding tissues. Second, an adaptive fusion strategy is used to fuse complementary predictions, which address the significant differences within the lesions caused by color heterogeneity. Third, a structural similarity loss with color information is proposed to measure the detail error between predicted lesions and truth lesions. Additionally, a PWS clinical dataset consisting of 1413 image pairs was established for the development and evaluation of PWS segmentation algorithms. To verify the effectiveness and superiority of the proposed method, we compared it with other state-of-the-art methods on our collected dataset and four publicly available skin lesion datasets (ISIC 2016, ISIC 2017, ISIC 2018, and PH2). The experimental results show that our method achieves remarkable performance in comparison with other state-of-the-art methods on our collected dataset, achieving 92.29% and 86.14% on Dice and Jaccard metrics, respectively. Comparative experiments on other datasets also confirmed the reliability and potential capability of M-CSAFN in skin lesion segmentation.

STNet: shape and texture joint learning through two-stream network for knowledge-guided image recognition

Introduction

The human brain processes shape and texture information separately through different neurons in the visual system. In intelligent computer-aided imaging diagnosis, pre-trained feature extractors are commonly used in various medical image recognition methods, common pre-training datasets such as ImageNet tend to improve the texture representation of the model but make it ignore many shape features. Weak shape feature representation is disadvantageous for some tasks that focus on shape features in medical image analysis.

Methods

Inspired by the function of neurons in the human brain, in this paper, we proposed a shape-and-texture-biased two-stream network to enhance the shape feature representation in knowledge-guided medical image analysis. First, the two-stream network shape-biased stream and a texture-biased stream are constructed through classification and segmentation multi-task joint learning. Second, we propose pyramid-grouped convolution to enhance the texture feature representation and introduce deformable convolution to enhance the shape feature extraction. Third, we used a channel-attention-based feature selection module in shape and texture feature fusion to focus on the key features and eliminate information redundancy caused by feature fusion. Finally, aiming at the problem of model optimization difficulty caused by the imbalance in the number of benign and malignant samples in medical images, an asymmetric loss function was introduced to improve the robustness of the model.

Results and conclusion

We applied our method to the melanoma recognition task on ISIC-2019 and XJTU-MM datasets, which focus on both the texture and shape of the lesions. The experimental results on dermoscopic image recognition and pathological image recognition datasets show the proposed method outperforms the compared algorithms and prove the effectiveness of our method.

A survey on deep learning for skin lesion segmentation

Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 177 research papers that deal with deep learning-based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. To facilitate comparisons, we summarize all examined works in a comprehensive table as well as an interactive table available online³.

Multiclass skin lesion localization and classification using deep learning based features fusion and selection framework for smart healthcare

BACKGROUND

The idea of smart healthcare has gradually gained attention as a result of the information technology industry's rapid development. Smart healthcare uses next-generation technologies i.e., artificial intelligence (AI) and Internet of Things (IoT), to intelligently transform current medical methods to make them more efficient, dependable and individualized. One of the most prominent uses of telemedicine and e-health in medical image analysis is teledermatology. Telecommunications technologies are used in this industry to send medical information to professionals. Teledermatology is a useful method for the identification of skin lesions, particularly in rural locations, because the skin is visually perceptible. One of the most recent tools for diagnosing skin cancer is dermoscopy. To classify skin malignancies, numerous computational approaches have been proposed in the literature. However, difficulties still exist i.e., lesions with low contrast, imbalanced datasets, high level of memory complexity, and the extraction of redundant features.

METHODS

In this work, a unified CAD model is proposed based on a deep learning framework for skin lesion segmentation and classification. In the proposed approach, the source dermoscopic images are initially pre-processed using a contrast enhancement based modified bio-inspired multiple exposure fusion approach. In the second stage, a custom 26-layered convolutional neural network (CNN) architecture is designed to segment the skin lesion regions. In the third stage, four pre-trained CNN models (Xception, ResNet-50, ResNet-101 and VGG16) are modified and trained using transfer learning on the segmented lesion images. In the fourth stage, the deep features vectors are extracted from all the CNN models and fused using the convolutional sparse image decomposition fusion approach. In the fifth stage, the univariate measurement and Poisson distribution feature selection approach is used for the best features selection for classification. Finally, the selected features are fed to the multi-class support vector machine (MC-SVM) for the final classification.

RESULTS

The proposed approach employed to the HAM10000, ISIC2018, ISIC2019, and PH2 datasets and achieved an accuracy of 98.57%, 98.62%, 93.47%, and 98.98% respectively which are better than previous works.

CONCLUSION

When compared to renowned state-of-the-art methods, experimental results show that the proposed skin lesion detection and classification approach achieved higher performance in terms of both visually and enhanced quantitative evaluation with enhanced accuracy.

SSD-KD: A self-supervised diverse knowledge distillation method for lightweight skin lesion classification using dermoscopic images

Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Over the last few years, computer-aided diagnosis has been rapidly developed and make great progress in healthcare and medical practices due to the advances in artificial intelligence, particularly with the adoption of convolutional neural networks. However, most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. In this case, the knowledge distillation (KD) method has been proven as an efficient tool to help improve the adaptability of lightweight models under limited resources, meanwhile keeping a high-level representation capability. To bridge the gap, this study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin disease classification. Our method models an intra-instance relational feature representation and integrates it with existing KD research. A dual relational knowledge distillation architecture is self-supervised trained while the weighted softened outputs are also exploited to enable the student model to capture richer knowledge from the teacher model. To demonstrate the effectiveness of our method, we conduct experiments on ISIC 2019, a large-scale open-accessed benchmark of skin diseases dermoscopic images. Experiments show that our distilled MobileNetV2 can achieve an accuracy as high as 85% for the classification tasks of 8 different skin diseases with minimal parameters and computing requirements. Ablation studies confirm the effectiveness of our intra- and inter-instance relational knowledge integration strategy. Compared with state-of-the-art knowledge distillation techniques, the proposed method demonstrates improved performance. To the best of our knowledge, this is the first deep knowledge distillation application for multi-disease classification on the large-scale dermoscopy database. Our codes and models are available at https://github.com/enkiwang/Portable-Skin-Lesion-Diagnosis.

Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study

BACKGROUND

Dermoscopy is commonly used for the evaluation of pigmented lesions, but agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labeling of medical data is a bottleneck in the development of machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images.

OBJECTIVE

The aim of this study is to demonstrate that crowdsourcing can be used to label basic dermoscopic structures from images of pigmented lesions with similar reliability to a group of experts.

METHODS

First, we obtained labels of 248 images of melanocytic lesions with 31 dermoscopic "subfeatures" labeled by 20 dermoscopy experts. These were then collapsed into 6 dermoscopic "superfeatures" based on structural similarity, due to low interrater reliability (IRR): dots, globules, lines, network structures, regression structures, and vessels. These images were then used as the gold standard for the crowd study. The commercial platform DiagnosUs was used to obtain annotations from a nonexpert crowd for the presence or absence of the 6 superfeatures in each of the 248 images. We replicated this methodology with a group of 7 dermatologists to allow direct comparison with the nonexpert crowd. The Cohen κ value was used to measure agreement across raters.

RESULTS

In total, we obtained 139,731 ratings of the 6 dermoscopic superfeatures from the crowd. There was relatively lower agreement for the identification of dots and globules (the median κ values were 0.526 and 0.395, respectively), whereas network structures and vessels showed the highest agreement (the median κ values were 0.581 and 0.798, respectively). This pattern was also seen among the expert raters, who had median κ values of 0.483 and 0.517 for dots and globules, respectively, and 0.758 and 0.790 for network structures and vessels. The median κ values between nonexperts and thresholded average-expert readers were 0.709 for dots, 0.719 for globules, 0.714 for lines, 0.838 for network structures, 0.818 for regression structures, and 0.728 for vessels.

CONCLUSIONS

This study confirmed that IRR for different dermoscopic features varied among a group of experts; a similar pattern was observed in a nonexpert crowd. There was good or excellent agreement for each of the 6 superfeatures between the crowd and the experts, highlighting the similar reliability of the crowd for labeling dermoscopic images. This confirms the feasibility and dependability of using crowdsourcing as a scalable solution to annotate large sets of dermoscopic images, with several potential clinical and educational applications, including the development of novel, explainable ML tools.

Developments and Clinical Applications of Noninvasive Optical Technologies for Skin Cancer Diagnosis

Skin cancer has shown a sharp increase in prevalence over the past few decades and currently accounts for one-third of all cancers diagnosed. The most lethal form of skin cancer is melanoma, which develops in 4% of individuals. The rising prevalence and increased number of fatalities of skin cancer put a significant burden on healthcare resources and the economy. However, early detection and treatment greatly improve survival rates for patients with skin cancer. Since the rising rates of both the incidence and mortality have been particularly noticeable with melanoma, significant resources have been allocated to research aimed at earlier diagnosis and a deeper knowledge of the disease. Dermoscopy, reflectance confocal microscopy, optical coherence tomography, multiphoton-excited fluorescence imaging, and dermatofluorescence are only a few of the optical modalities reviewed here that have been employed to enhance noninvasive diagnosis of skin cancer in recent years. This review article discusses the methodology behind newly emerging noninvasive optical diagnostic technologies, their clinical applications, and advantages and disadvantages of these techniques, as well as the potential for their further advancement in the future.

Skin Lesion Area Segmentation Using Attention Squeeze U-Net for Embedded Devices

Melanoma is the deadliest form of skin cancer. Early diagnosis of malignant lesions is crucial for reducing mortality. The use of deep learning techniques on dermoscopic images can help in keeping track of the change over time in the appearance of the lesion, which is an important factor for detecting malignant lesions. In this paper, we present a deep learning architecture called Attention Squeeze U-Net for skin lesion area segmentation specifically designed for embedded devices. The main goal is to increase the patient empowerment through the adoption of deep learning algorithms that can run locally on smartphones or low cost embedded devices. This can be the basis to (1) create a history of the lesion, (2) reduce patient visits to the hospital, and (3) protect the privacy of the users. Quantitative results on publicly available data demonstrate that it is possible to achieve good segmentation results even with a compact model.

Slit lamp polarized dermoscopy: a cost-effective tool to assess eyelid lesions

PURPOSE

Dermoscopy is a complementary examination of skin lesions, which allows the observation of anatomical features invisible to the naked eye. Its use increases the diagnostic accuracy of skin tumors. The development of polarized dermoscopy allowed the observation of deeper skin structures, without the need of skin contact. The purpose of this study was to present a low-cost prototype through the adaptation of polarized lenses on a slit lamp in order to assess anatomical aspects invisible to conventional biomicroscopy in eyelid lesions.

METHODS

Twenty two eyelid lesions were documented using a prototype, compound of two polarizing filters, orthogonal to each other, adapted to a slit lamp with an integrated digital camera. Images of the eyelid lesions were also obtained with non-polarized biomicroscopy and with a portable dermatoscope, and were compared regarding anatomical aspects.

RESULTS

Anatomical structures imperceptible to conventional ophthalmic examination were evidenced using the polarized lenses, demonstrating that this tool can be useful to the ophthalmologist when assessing eyelid lesions. We have obtained high-quality images of the lesions. The slit lamp provided higher magnification, better focus control and easier assessment of eyelid lesions than the portable dermatoscope.

CONCLUSION

Ophthalmologists already use the slit lamp in their practice. The adaptation of polarized lenses to this device is a cost-effective, fast and non-invasive method that permits to improve the diagnostic accuracy of eyelid lesions, evidencing anatomical structures imperceptible to conventional ophthalmic examination.

MDFNet: application of multimodal fusion method based on skin image and clinical data to skin cancer classification

PURPOSE

Skin cancer is one of the ten most common cancer types in the world. Early diagnosis and treatment can effectively reduce the mortality of patients. Therefore, it is of great significance to develop an intelligent diagnosis system for skin cancer. According to the survey, at present, most intelligent diagnosis systems of skin cancer only use skin image data, but the multi-modal cross-fusion analysis using image data and patient clinical data is limited. Therefore, to further explore the complementary relationship between image data and patient clinical data, we propose multimode data fusion diagnosis network (MDFNet), a framework for skin cancer based on data fusion strategy.

METHODS

MDFNet establishes an effective mapping among heterogeneous data features, effectively fuses clinical skin images and patient clinical data, and effectively solves the problems of feature paucity and insufficient feature richness that only use single-mode data.

RESULTS

The experimental results present that our proposed smart skin cancer diagnosis model has an accuracy of 80.42%, which is an improvement of about 9% compared with the model accuracy using only medical images, thus effectively confirming the unique fusion advantages exhibited by MDFNet.

CONCLUSIONS

This illustrates that MDFNet can not only be applied as an effective auxiliary diagnostic tool for skin cancer diagnosis, help physicians improve clinical decision-making ability and effectively improve the efficiency of clinical medicine diagnosis, but also its proposed data fusion method fully exerts the advantage of information convergence and has a certain reference value for the intelligent diagnosis of numerous clinical diseases.

Clinically Inspired Skin Lesion Classification through the Detection of Dermoscopic Criteria for Basal Cell Carcinoma

Background and Objective. Skin cancer is the most common cancer worldwide. One of the most common non-melanoma tumors is basal cell carcinoma (BCC), which accounts for 75% of all skin cancers. There are many benign lesions that can be confused with these types of cancers, leading to unnecessary biopsies. In this paper, a new method to identify the different BCC dermoscopic patterns present in a skin lesion is presented. In addition, this information is applied to classify skin lesions into BCC and non-BCC. Methods. The proposed method combines the information provided by the original dermoscopic image, introduced in a convolutional neural network (CNN), with deep and handcrafted features extracted from color and texture analysis of the image. This color analysis is performed by transforming the image into a uniform color space and into a color appearance model. To demonstrate the validity of the method, a comparison between the classification obtained employing exclusively a CNN with the original image as input and the classification with additional color and texture features is presented. Furthermore, an exhaustive comparison of classification employing different color and texture measures derived from different color spaces is presented. Results. Results show that the classifier with additional color and texture features outperforms a CNN whose input is only the original image. Another important achievement is that a new color cooccurrence matrix, proposed in this paper, improves the results obtained with other texture measures. Finally, sensitivity of 0.99, specificity of 0.94 and accuracy of 0.97 are achieved when lesions are classified into BCC or non-BCC. Conclusions. To the best of our knowledge, this is the first time that a methodology to detect all the possible patterns that can be present in a BCC lesion is proposed. This detection leads to a clinically explainable classification into BCC and non-BCC lesions. In this sense, the classification of the proposed tool is based on the detection of the dermoscopic features that dermatologists employ for their diagnosis.

Fusion of Local and Global Feature Representation With Sparse Autoencoder for Improved Melanoma Classification

Automated skin cancer diagnosis is challenging due to inter-class uniformity, intra-class variation, and the complex structure of dermoscopy images. Convolutional neural networks (CNN) have recently made considerable progress in melanoma classification, even in the presence of limited skin images. One of the drawbacks of these methods is the loss of image details caused by downsampling high-resolution skin images to a low resolution. Further, most approaches extract features only from the whole skin image. This paper proposes an ensemble feature fusion and sparse autoencoder (SAE) based framework to overcome the above issues and improve melanoma classification performance. The proposed method extracts features from two streams, local and global, using a pre-trained CNN model. The local stream extracts features from image patches, while the global stream derives features from the whole skin image, preserving both local and global representation. The features are then fused, and an SAE framework is subsequently designed to enrich the feature representation further. The proposed method is validated on ISIC 2016 dataset and the experimental results indicate the superiority of the proposed approach.

Detection of Skin Cancer Based on Skin Lesion Images Using Deep Learning

An increasing number of genetic and metabolic anomalies have been determined to lead to cancer, generally fatal. Cancerous cells may spread to any body part, where they can be life-threatening. Skin cancer is one of the most common types of cancer, and its frequency is increasing worldwide. The main subtypes of skin cancer are squamous and basal cell carcinomas, and melanoma, which is clinically aggressive and responsible for most deaths. Therefore, skin cancer screening is necessary. One of the best methods to accurately and swiftly identify skin cancer is using deep learning (DL). In this research, the deep learning method convolution neural network (CNN) was used to detect the two primary types of tumors, malignant and benign, using the ISIC2018 dataset. This dataset comprises 3533 skin lesions, including benign, malignant, nonmelanocytic, and melanocytic tumors. Using ESRGAN, the photos were first retouched and improved. The photos were augmented, normalized, and resized during the preprocessing step. Skin lesion photos could be classified using a CNN method based on an aggregate of results obtained after many repetitions. Then, multiple transfer learning models, such as Resnet50, InceptionV3, and Inception Resnet, were used for fine-tuning. In addition to experimenting with several models (the designed CNN, Resnet50, InceptionV3, and Inception Resnet), this study’s innovation and contribution are the use of ESRGAN as a preprocessing step. Our designed model showed results comparable to the pretrained model. Simulations using the ISIC 2018 skin lesion dataset showed that the suggested strategy was successful. An 83.2% accuracy rate was achieved by the CNN, in comparison to the Resnet50 (83.7%), InceptionV3 (85.8%), and Inception Resnet (84%) models.

Colored Texture Analysis Fuzzy Entropy Methods with a Dermoscopic Application

Texture analysis is a subject of intensive focus in research due to its significant role in the field of image processing. However, few studies focus on colored texture analysis and even fewer use information theory concepts. Entropy measures have been proven competent for gray scale images. However, to the best of our knowledge, there are no well-established entropy methods that deal with colored images yet. Therefore, we propose the recent colored bidimensional fuzzy entropy measure, FuzEnC2D, and introduce its new multi-channel approaches, FuzEnV2D and FuzEnM2D, for the analysis of colored images. We investigate their sensitivity to parameters and ability to identify images with different irregularity degrees, and therefore different textures. Moreover, we study their behavior with colored Brodatz images in different color spaces. After verifying the results with test images, we employ the three methods for analyzing dermoscopic images of malignant melanoma and benign melanocytic nevi. FuzEnC2D, FuzEnV2D, and FuzEnM2D illustrate a good differentiation ability between the two-similar in appearance-pigmented skin lesions. The results outperform those of a well-known texture analysis measure. Our work provides the first entropy measure studying colored images using both single and multi-channel approaches.

An Effective Skin Cancer Classification Mechanism via Medical Vision Transformer

Skin Cancer (SC) is considered the deadliest disease in the world, killing thousands of people every year. Early SC detection can increase the survival rate for patients up to 70%, hence it is highly recommended that regular head-to-toe skin examinations are conducted to determine whether there are any signs or symptoms of SC. The use of Machine Learning (ML)-based methods is having a significant impact on the classification and detection of SC diseases. However, there are certain challenges associated with the accurate classification of these diseases such as a lower detection accuracy, poor generalization of the models, and an insufficient amount of labeled data for training. To address these challenges, in this work we developed a two-tier framework for the accurate classification of SC. During the first stage of the framework, we applied different methods for data augmentation to increase the number of image samples for effective training. As part of the second tier of the framework, taking into consideration the promising performance of the Medical Vision Transformer (MVT) in the analysis of medical images, we developed an MVT-based classification model for SC. This MVT splits the input image into image patches and then feeds these patches to the transformer in a sequence structure, like word embedding. Finally, Multi-Layer Perceptron (MLP) is used to classify the input image into the corresponding class. Based on the experimental results achieved on the Human Against Machine (HAM10000) datasets, we concluded that the proposed MVT-based model achieves better results than current state-of-the-art techniques for SC classification.

Skin Lesion Analysis for Melanoma Detection Using the Novel Deep Learning Model Fuzzy GC-SCNN

Melanoma is easily detectable by visual examination since it occurs on the skin’s surface. In melanomas, which are the most severe types of skin cancer, the cells that make melanin are affected. However, the lack of expert opinion increases the processing time and cost of computer-aided skin cancer detection. As such, we aimed to incorporate deep learning algorithms to conduct automatic melanoma detection from dermoscopic images. The fuzzy-based GrabCut-stacked convolutional neural networks (GC-SCNN) model was applied for image training. The image features extraction and lesion classification were performed on different publicly available datasets. The fuzzy GC-SCNN coupled with the support vector machines (SVM) produced 99.75% classification accuracy and 100% sensitivity and specificity, respectively. Additionally, model performance was compared with existing techniques and outcomes suggesting the proposed model could detect and classify the lesion segments with higher accuracy and lower processing time than other techniques.

Hair removal in dermoscopy images using variational autoencoders

BACKGROUND

In recent years, melanoma is rising at a faster rate compared to other cancers. Although it is the most serious type of skin cancer, the diagnosis at early stages makes it curable. Dermoscopy is a reliable medical technique used to detect melanoma by using a dermoscope to examine the skin. In the last few decades, digital imaging devices have made great progress which allowed capturing and storing high-quality images from these examinations. The stored images are now being standardized and used for the automatic detection of melanoma. However, when the hair covers the skin, this makes the task challenging. Therefore, it is important to eliminate the hair to get accurate results.

METHODS

In this paper, we propose a simple yet efficient method for hair removal using a variational autoencoder without the need for paired samples. The encoder takes as input a dermoscopy image and builds a latent distribution that ignores hair as it is considered noise, while the decoder reconstructs a hair-free image. Both encoder and decoder use a decent convolutional neural networks architecture that provides high performance. The construction of our model comprises two stages of training. In the first stage, the model has trained on hair-occluded images to output hair-free images, and in the second stage, it is optimized using hair-free images to preserve the image textures. Although the variational autoencoder produces hair-free images, it does not maintain the quality of the generated images. Thus, we explored the use of three-loss functions including the structural similarity index (SSIM), L1-norm, and L2-norm to improve the visual quality of the generated images.

RESULTS

The evaluation of the hair-free reconstructed images is carried out using t-distributed stochastic neighbor embedding (SNE) feature mapping by visualizing the distribution of the real hair-free images and the synthesized hair-free images. The conducted experiments on the publicly available dataset HAM10000 show that our method is very efficient.

Efficient structural pseudoinverse learning-based hierarchical representation learning for skin lesion classification

The success of deep learning in skin lesion classification mainly depends on the ultra-deep neural network and the significantly large training data set. Deep learning training is usually time-consuming, and large datasets with labels are hard to obtain, especially skin lesion images. Although pre-training and data augmentation can alleviate these issues, there are still some problems: (1) the data domain is not consistent, resulting in the slow convergence; and (2) low robustness to confusing skin lesions. To solve these problems, we propose an efficient structural pseudoinverse learning-based hierarchical representation learning method. Preliminary feature extraction, shallow network feature extraction and deep learning feature extraction are carried out respectively before the classification of skin lesion images. Gabor filter and pre-trained deep convolutional neural network are used for preliminary feature extraction. The structural pseudoinverse learning (S-PIL) algorithm is used to extract the shallow features. Then, S-PIL preliminarily identifies the skin lesion images that are difficult to be classified to form a new training set for deep learning feature extraction. Through the hierarchical representation learning, we analyze the features of skin lesion images layer by layer to improve the final classification. Our method not only avoid the slow convergence caused by inconsistency of data domain but also enhances the training of confusing examples. Without using additional data, our approach outperforms existing methods in the ISIC 2017 and ISIC 2018 datasets.

MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation

Methods based on convolutional neural networks have improved the performance of biomedical image segmentation. However, most of these methods cannot efficiently segment objects of variable sizes and train on small and biased datasets, which are common for biomedical use cases. While methods exist that incorporate multi-scale fusion approaches to address the challenges arising with variable sizes, they usually use complex models that are more suitable for general semantic segmentation problems. In this paper, we propose a novel architecture called Multi-Scale Residual Fusion Network (MSRF-Net), which is specially designed for medical image segmentation. The proposed MSRF-Net is able to exchange multi-scale features of varying receptive fields using a Dual-Scale Dense Fusion (DSDF) block. Our DSDF block can exchange information rigorously across two different resolution scales, and our MSRF sub-network uses multiple DSDF blocks in sequence to perform multi-scale fusion. This allows the preservation of resolution, improved information flow and propagation of both high- and low-level features to obtain accurate segmentation maps. The proposed MSRF-Net allows to capture object variabilities and provides improved results on different biomedical datasets. Extensive experiments on MSRF-Net demonstrate that the proposed method outperforms the cutting edge medical image segmentation methods on four publicly available datasets. We achieve the Dice Coefficient (DSC) of 0.9217, 0.9420, and 0.9224, 0.8824 on Kvasir-SEG, CVC-ClinicDB, 2018 Data Science Bowl dataset, and ISIC-2018 skin lesion segmentation challenge dataset respectively. We further conducted generalizability tests that also achieved the highest DSC score with 0.7921 and 0.7575 on CVC-ClinicDB and Kvasir-SEG, respectively.

Characterizing Malignant Melanoma Clinically Resembling Seborrheic Keratosis Using Deep Knowledge Transfer

Simple Summary

Malignant melanomas (MMs) with aypical clinical presentation constitute a diagnostic pitfall, and false negatives carry the risk of a diagnostic delay and improper disease management. Among the most common, challenging presentation forms of MMs are those that clinically resemble seborrheic keratosis (SK). On the other hand, SK may mimic melanoma, producing ‘false positive overdiagnosis’ and leading to needless excisions. The evolving efficiency of deep learning algorithms in image recognition and the availability of large image databases have accelerated the development of advanced computer-aided systems for melanoma detection. In the present study, we used image data from the International Skin Image Collaboration archive to explore the capacity of deep knowledge transfer in the challenging diagnostic task of the atypical skin tumors of MM and SK.

Abstract

Malignant melanomas resembling seborrheic keratosis (SK-like MMs) are atypical, challenging to diagnose melanoma cases that carry the risk of delayed diagnosis and inadequate treatment. On the other hand, SK may mimic melanoma, producing a ‘false positive’ with unnecessary lesion excisions. The present study proposes a computer-based approach using dermoscopy images for the characterization of SΚ-like MMs. Dermoscopic images were retrieved from the International Skin Imaging Collaboration archive. Exploiting image embeddings from pretrained convolutional network VGG16, we trained a support vector machine (SVM) classification model on a data set of 667 images. SVM optimal hyperparameter selection was carried out using the Bayesian optimization method. The classifier was tested on an independent data set of 311 images with atypical appearance: MMs had an absence of pigmented network and had an existence of milia-like cysts. SK lacked milia-like cysts and had a pigmented network. Atypical MMs were characterized with a sensitivity and specificity of 78.6% and 84.5%, respectively. The advent of deep learning in image recognition has attracted the interest of computer science towards improved skin lesion diagnosis. Open-source, public access archives of skin images empower further the implementation and validation of computer-based systems that might contribute significantly to complex clinical diagnostic problems such as the characterization of SK-like MMs.

Multi-features extraction based on deep learning for skin lesion classification

For various forms of skin lesion, many different feature extraction methods have been investigated so far. Indeed, feature extraction is a crucial step in machine learning processes. In general, we can distinct handcrafted and deep learning features. In this paper, we investigate the efficiency of using 17 commonly pre-trained convolutional neural networks (CNN) architectures as feature extractors and of 24 machine learning classifiers to evaluate the classification of skin lesions from two different datasets: ISIC 2019 and PH2. In this research, we find out that a DenseNet201 combined with Fine KNN or Cubic SVM achieved the best results in accuracy (92.34% and 91.71%) for the ISIC 2019 dataset. The results also show that the suggested method outperforms others approaches with an accuracy of 99% on the PH2 dataset.

Incorporating clinical knowledge with constrained classifier chain into a multimodal deep network for melanoma detection

In recent years, vast developments in Computer-Aided Diagnosis (CAD) for skin diseases have generated much interest from clinicians and other eventual end-users of this technology. Introducing clinical domain knowledge to these machine learning strategies can help dispel the black box nature of these tools, strengthening clinician trust. Clinical domain knowledge also provides new information channels which can improve CAD diagnostic performance. In this paper, we propose a novel framework for malignant melanoma (MM) detection by fusing clinical images and dermoscopic images. The proposed method combines a multi-labeled deep feature extractor and clinically constrained classifier chain (CC). This allows the 7-point checklist, a clinician diagnostic algorithm, to be included in the decision level while maintaining the clinical importance of the major and minor criteria in the checklist. Our proposed framework achieved an average accuracy of 81.3% for detecting all criteria and melanoma when testing on a publicly available 7-point checklist dataset. This is the highest reported results, outperforming state-of-the-art methods in the literature by 6.4% or more. Analyses also show that the proposed system surpasses the single modality system of using either clinical images or dermoscopic images alone and the systems without adopting the approach of multi-label and clinically constrained classifier chain. Our carefully designed system demonstrates a substantial improvement over melanoma detection. By keeping the familiar major and minor criteria of the 7-point checklist and their corresponding weights, the proposed system may be more accepted by physicians as a human-interpretable CAD tool for automated melanoma detection.

Skin Melanoma Detection in Microscopic Images Using HMM-Based Asymmetric Analysis and Expectation Maximization

Melanoma is one of the deadliest types of skin cancer with increasing incidence. The most definitive diagnosis method is the histopathological examination of the tissue sample. In this paper, a melanoma detection algorithm is proposed based on decision-level fusion and a Hidden Markov Model (HMM), whose parameters are optimized using Expectation Maximization (EM) and asymmetric analysis. The texture heterogeneity of the samples is determined using asymmetric analysis. A fusion-based HMM classifier trained using EM is introduced. For this purpose, a novel texture feature is extracted based on two local binary patterns, namely local difference pattern (LDP) and statistical histogram features of the microscopic image. Extensive experiments demonstrate that the proposed melanoma detection algorithm yields a total error of less than 0.04%.

Skin Lesion Segmentation and Multiclass Classification Using Deep Learning Features and Improved Moth Flame Optimization

Manual diagnosis of skin cancer is time-consuming and expensive; therefore, it is essential to develop automated diagnostics methods with the ability to classify multiclass skin lesions with greater accuracy. We propose a fully automated approach for multiclass skin lesion segmentation and classification by using the most discriminant deep features. First, the input images are initially enhanced using local color-controlled histogram intensity values (LCcHIV). Next, saliency is estimated using a novel Deep Saliency Segmentation method, which uses a custom convolutional neural network (CNN) of ten layers. The generated heat map is converted into a binary image using a thresholding function. Next, the segmented color lesion images are used for feature extraction by a deep pre-trained CNN model. To avoid the curse of dimensionality, we implement an improved moth flame optimization (IMFO) algorithm to select the most discriminant features. The resultant features are fused using a multiset maximum correlation analysis (MMCA) and classified using the Kernel Extreme Learning Machine (KELM) classifier. The segmentation performance of the proposed methodology is analyzed on ISBI 2016, ISBI 2017, ISIC 2018, and PH2 datasets, achieving an accuracy of 95.38%, 95.79%, 92.69%, and 98.70%, respectively. The classification performance is evaluated on the HAM10000 dataset and achieved an accuracy of 90.67%. To prove the effectiveness of the proposed methods, we present a comparison with the state-of-the-art techniques.

Non-invasive optical methods for melanoma diagnosis

Cutaneous melanoma is one of the most common malignancies with increased incidence in the past few decades, making it a significant public health problem. The early diagnosis of melanoma is a major factor in improving patient's survival. The traditional pathway to melanoma diagnosis starts with a visual diagnosis, followed by subsequent biopsy and histopathologic evaluation. Recently, multiple innovative optical technology-based methods, including dermoscopy, reflectance confocal microscopy, optical coherence tomography, multiphoton excited fluorescence imaging and stepwise two-photon excited fluorescence (dermatofluoroscopy), have been developed to increase the diagnostic accuracy for the non-invasive melanoma diagnosis. Some of them have already been applied to real-life clinical settings, others require more research and development. These technologies show promise in facilitating the diagnosis of melanoma since they are non-invasive, sensitive, objective and easy to apply. Diagnostic accuracy, detection time, portability and the cost-effectiveness of the device are all aspects that need to be improved. This article reviews the method of these emerging optical non-invasive diagnostic technologies, their clinical application, their benefits and limitations, as well as their possible future development.

Leveraging the Bhattacharyya coefficient for uncertainty quantification in deep neural networks

Modern deep learning models achieve state-of-the-art results for many tasks in computer vision, such as image classification and segmentation. However, its adoption into high-risk applications, e.g. automated medical diagnosis systems, happens at a slow pace. One of the main reasons for this is that regular neural networks do not capture uncertainty. To assess uncertainty in classification, several techniques have been proposed casting neural network approaches in a Bayesian setting. Amongst these techniques, Monte Carlo dropout is by far the most popular. This particular technique estimates the moments of the output distribution through sampling with different dropout masks. The output uncertainty of a neural network is then approximated as the sample variance. In this paper, we highlight the limitations of such a variance-based uncertainty metric and propose an novel approach. Our approach is based on the overlap between output distributions of different classes. We show that our technique leads to a better approximation of the inter-class output confusion. We illustrate the advantages of our method using benchmark datasets. In addition, we apply our metric to skin lesion classification—a real-world use case—and show that this yields promising results.

Computer-aided classification of suspicious pigmented lesions using wide-field images

BACKGROUND AND OBJECTIVE

Early identification of melanoma is conducted through whole-body visual examinations to detect suspicious pigmented lesions, a situation that fluctuates in accuracy depending on the experience and time of the examiner. Computer-aided diagnosis tools for skin lesions are typically trained using pre-selected single-lesion images, taken under controlled conditions, which limits their use in wide-field scenes. Here, we propose a computer-aided classifier system with such input conditions to aid in the rapid identification of suspicious pigmented lesions at the primary care level.

METHODS

133 patients with a multitude of skin lesions were recruited for this study. All lesions were examined by a board-certified dermatologist and classified into "suspicious" and "non-suspicious". A new clinical database was acquired and created by taking Wide-Field images of all major body parts with a consumer-grade camera under natural illumination condition and with a consistent source of image variability. 3-8 images were acquired per patient on different sites of the body, and a total of 1759 pigmented lesions were extracted. A machine learning classifier was optimized and build into a computer aided classification system to binary classify each lesion using a suspiciousness score.

RESULTS

In a testing set, our computer-aided classification system achieved a sensitivity of 100% for suspicious pigmented lesions that were later confirmed by dermoscopy examination ("SPL_A") and 83.2% for suspicious pigmented lesions that were not confirmed after examination ("SPL_B"). Sensitivity for non-suspicious lesions was 72.1%, and accuracy was 75.9%. With these results we defined a suspiciousness score that is aligned with common macro-screening (naked eye) practices.

CONCLUSIONS

This work demonstrates that wide-field photography combined with computer-aided classification systems can distinguish suspicious from non-suspicious pigmented lesions, and might be effective to assess the severity of a suspicious pigmented lesions. We believe this approach could be useful to support skin screenings at a population-level.