Attention Residual Learning for Skin Lesion Classification

Trustworthy learning with (un)sure annotation for lung nodule diagnosis with CT

Recent evolution in deep learning has proven its value for CT-based lung nodule classification. Most current techniques are intrinsically black-box systems, suffering from two generalizability issues in clinical practice. First, benign-malignant discrimination is often assessed by human observers without pathologic diagnoses at the nodule level. We termed these data as "unsure-annotation data". Second, a classifier does not necessarily acquire reliable nodule features for stable learning and robust prediction with patch-level labels during learning. In this study, we construct a sure-annotation dataset with pathologically-confirmed labels and propose a collaborative learning framework to facilitate sure nodule classification by integrating unsure-annotation data knowledge through nodule segmentation and malignancy score regression. A loss function is designed to learn reliable features by introducing interpretability constraints regulated with nodule segmentation maps. Furthermore, based on model inference results that reflect the understanding from both machine and experts, we explore a new nodule analysis method for similar historical nodule retrieval and interpretable diagnosis. Detailed experimental results demonstrate that our approach is beneficial for achieving improved performance coupled with trustworthy model reasoning for lung cancer prediction with limited data. Extensive cross-evaluation results further illustrate the effect of unsure-annotation data for deep-learning based methods in lung nodule classification.

Toward Multicenter Skin Lesion Classification Using Deep Neural Network With Adaptively Weighted Balance Loss

Recently, deep neural network-based methods have shown promising advantages in accurately recognizing skin lesions from dermoscopic images. However, most existing works focus more on improving the network framework for better feature representation but ignore the data imbalance issue, limiting their flexibility and accuracy across multiple scenarios in multi-center clinics. Generally, different clinical centers have different data distributions, which presents challenging requirements for the network's flexibility and accuracy. In this paper, we divert the attention from framework improvement to the data imbalance issue and propose a new solution for multi-center skin lesion classification by introducing a novel adaptively weighted balance (AWB) loss to the conventional classification network. Benefiting from AWB, the proposed solution has the following advantages: 1) it is easy to satisfy different practical requirements by only changing the backbone; 2) it is user-friendly with no tuning on hyperparameters; and 3) it adaptively enables small intraclass compactness and pays more attention to the minority class. Extensive experiments demonstrate that, compared with solutions equipped with state-of-the-art loss functions, the proposed solution is more flexible and more competent for tackling the multi-center imbalanced skin lesion classification task with considerable performance on two benchmark datasets. In addition, the proposed solution is proved to be effective in handling the imbalanced gastrointestinal disease classification task and the imbalanced DR grading task. Code is available at https://github.com/Weipeishan2021.

Learning from dermoscopic images in association with clinical metadata for skin lesion segmentation and classification

Automatic segmentation and classification of lesions are two clinically significant tasks in the computer-aided diagnosis of skin diseases. Both tasks are challenging due to the nonnegligible lesion differences in dermoscopic images from different patients. In this paper, we propose a novel pipeline to efficiently implement skin lesions' segmentation and classification tasks, which consists of a segmentation network and a classification network. To improve the performance of the segmentation network, we propose a novel module of Multi-Scale Holistic Feature Exploration (MSH) to thoroughly exploit perceptual clues latent among multi-scale feature maps as synthesized by the decoder. The MSH module enables holistic exploration of features across multiple scales to more effectively support downstream image analysis tasks. To boost the performance of the classification network, we propose a novel module of Cross-Modality Collaborative Feature Exploration (CMC) to discover latent discriminative features by collaboratively exploiting potential relationships between cross-modal features of dermoscopic images and clinical metadata. The CMC module enables dynamically capturing versatile interaction effects among cross-modal features during the model's representation learning procedure by discriminatively and adaptively learning the interaction weight associated with each crossmodality feature pair. In addition, to effectively reduce background noise and boost the lesion discrimination ability of the classification network, we crop the images based on lesion masks generated by the best segmentation model. We evaluate the proposed pipeline on the four public skin lesion datasets, where the ISIC 2018 and PH2 are for segmentation, and the ISIC 2019 and ISIC 2020 are combined into a new dataset, ISIC 2019&2020, for classification. It achieves a Jaccard index of 83.31% and 90.14% in skin lesion segmentation, an AUC of 97.98% and an Accuracy of 92.63% in skin lesion classification, which is superior to the performance of representative state-of-the-art skin lesion segmentation and classification methods. Last but not least, the new model for segmentation utilizes much fewer model parameters (3.3 M) than its peer approaches, leading to a greatly reduced number of labeled samples required for model training, which obtains substantially stronger robustness than its peers.

Drug-Resistant Tuberculosis Treatment Recommendation, and Multi-Class Tuberculosis Detection and Classification Using Ensemble Deep Learning-Based System

This research develops the TB/non-TB detection and drug-resistant categorization diagnosis decision support system (TB-DRC-DSS). The model is capable of detecting both TB-negative and TB-positive samples, as well as classifying drug-resistant strains and also providing treatment recommendations. The model is developed using a deep learning ensemble model with the various CNN architectures. These architectures include EfficientNetB7, mobileNetV2, and Dense-Net121. The models are heterogeneously assembled to create an effective model for TB-DRC-DSS, utilizing effective image segmentation, augmentation, and decision fusion techniques to improve the classification efficacy of the current model. The web program serves as the platform for determining if a patient is positive or negative for tuberculosis and classifying several types of drug resistance. The constructed model is evaluated and compared to current methods described in the literature. The proposed model was assessed using two datasets of chest X-ray (CXR) images collected from the references. This collection of datasets includes the Portal dataset, the Montgomery County dataset, the Shenzhen dataset, and the Kaggle dataset. Seven thousand and eight images exist across all datasets. The dataset was divided into two subsets: the training dataset (80%) and the test dataset (20%). The computational result revealed that the classification accuracy of DS-TB against DR-TB has improved by an average of 43.3% compared to other methods. The categorization between DS-TB and MDR-TB, DS-TB and XDR-TB, and MDR-TB and XDR-TB was more accurate than with other methods by an average of 28.1%, 6.2%, and 9.4%, respectively. The accuracy of the embedded multiclass model in the web application is 92.6% when evaluated with the test dataset, but 92.8% when evaluated with a random subset selected from the aggregate dataset. In conclusion, 31 medical staff members have evaluated and utilized the online application, and the final user preference score for the web application is 9.52 out of a possible 10.

RAMRU-CAM: Residual-Atrous MultiResUnet with Channel Attention Mechanism for cell segmentation

The task of cell segmentation in microscope images is difficult and popular. In recent years, deep learning-based techniques have made incredible progress in medical and microscopy image segmentation applications. In this paper, we propose a novel deep learning approach called Residual-Atrous MultiResUnet with Channel Attention Mechanism (RAMRU-CAM) for cell segmentation, which combines MultiResUnet architecture with Channel Attention Mechanism (CAM) and Residual-Atrous connections. The Residual-Atrous path mitigates the semantic gap between the encoder and decoder stages and manages the spatial dimension of feature maps. Furthermore, the Channel Attention Mechanism (CAM) blocks are used in the decoder stages to better maintain the spatial details before concatenating the feature maps from the encoder phases to the decoder phases. We evaluated our proposed model on the PhC-C2DH-U373 and Fluo-N2DH-GOWT1 datasets. The experimental results show that our proposed model outperforms recent variants of the U-Net model and the state-of-the-art approaches. We have demonstrated how our model can segment cells precisely while using fewer parameters and low computational complexity.

A Survey on Computer-Aided Intelligent Methods to Identify and Classify Skin Cancer

Melanoma is one of the skin cancer types that is more dangerous to human society. It easily spreads to other parts of the human body. An early diagnosis is necessary for a higher survival rate. Computer-aided diagnosis (CAD) is suitable for providing precise findings before the critical stage. The computer-aided diagnostic process includes preprocessing, segmentation, feature extraction, and classification. This study discusses the advantages and disadvantages of various computer-aided algorithms. It also discusses the current approaches, problems, and various types of datasets for skin images. Information about possible future works is also highlighted in this paper. The inferences derived from this survey will be useful for researchers carrying out research in skin cancer image analysis.

GraVIS: Grouping Augmented Views From Independent Sources for Dermatology Analysis

Self-supervised representation learning has been extremely successful in medical image analysis, as it requires no human annotations to provide transferable representations for downstream tasks. Recent self-supervised learning methods are dominated by noise-contrastive estimation (NCE, also known as contrastive learning), which aims to learn invariant visual representations by contrasting one homogeneous image pair with a large number of heterogeneous image pairs in each training step. Nonetheless, NCE-based approaches still suffer from one major problem that is one homogeneous pair is not enough to extract robust and invariant semantic information. Inspired by the archetypical triplet loss, we propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images, to group homogeneous dermatology images while separating heterogeneous ones. In addition, a hardness-aware attention is introduced and incorporated to address the importance of homogeneous image views with similar appearance instead of those dissimilar homogeneous ones. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks, sometimes by 5 percents under extremely limited supervision. More importantly, when equipped with the pre-trained weights provided by GraVIS, a single model could achieve better results than winners that heavily rely on ensemble strategies in the well-known ISIC 2017 challenge. Code is available at https://bit.ly/3xiFyjx.

A rotation meanout network with invariance for dermoscopy image classification and retrieval

The computer-aided diagnosis (CAD) system can provide a reference basis for the clinical diagnosis of skin diseases. Convolutional neural networks (CNNs) can not only extract visual elements such as colors and shapes but also semantic features. As such they have made great improvements in many tasks of dermoscopy images. The imaging of dermoscopy has no principal orientation, indicating that there are a large number of skin lesion rotations in the datasets. However, CNNs lack rotation invariance, which is bound to affect the robustness of CNNs against rotations. To tackle this issue, we propose a rotation meanout (RM) network to extract rotation-invariant features from dermoscopy images. In RM, each set of rotated feature maps corresponds to a set of outputs of the weight-sharing convolutions and they are fused using meanout strategy to obtain the final feature maps. Through theoretical derivation, the proposed RM network is rotation-equivariant and can extract rotation-invariant features when followed by the global average pooling (GAP) operation. The extracted rotation-invariant features can better represent the original data in classification and retrieval tasks for dermoscopy images. The RM is a general operation, which does not change the network structure or increase any parameters, and can be flexibly embedded in any part of CNNs. Extensive experiments are conducted on a dermoscopy image dataset. The results show that our method outperforms other anti-rotation methods and achieves great improvements in skin disease classification and retrieval tasks, indicating the potential of rotation invariance in the field of dermoscopy images.

Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis

Skin lesion diagnosis is a key step for skin cancer screening, which requires high accuracy and interpretability. Though many computer-aided methods, especially deep learning methods, have made remarkable achievements in skin lesion diagnosis, their generalization and interpretability are still a challenge. To solve this issue, we propose an interpretability-based multimodal convolutional neural network (IM-CNN), which is a multiclass classification model with skin lesion images and metadata of patients as input for skin lesion diagnosis. The structure of IM-CNN consists of three main paths to deal with metadata, features extracted from segmented skin lesion with domain knowledge, and skin lesion images, respectively. We add interpretable visual modules to provide explanations for both images and metadata. In addition to area under the ROC curve (AUC), sensitivity, and specificity, we introduce a new indicator, an AUC curve with a sensitivity larger than 80% (AUC_SEN_80) for performance evaluation. Extensive experimental studies are conducted on the popular HAM10000 dataset, and the results indicate that the proposed model has overwhelming advantages compared with popular deep learning models, such as DenseNet, ResNet, and other state-of-the-art models for melanoma diagnosis. The proposed multimodal model also achieves on average 72% and 21% improvement in terms of sensitivity and AUC_SEN_80, respectively, compared with the single-modal model. The visual explanations can also help gain trust from dermatologists and realize man-machine collaborations, effectively reducing the limitation of black-box models in supporting medical decision making.

Headache classification and automatic biomarker extraction from structural MRIs using deep learning

Data-driven machine-learning methods on neuroimaging (e.g. MRI) are of great interest for the investigation and classification of neurological diseases. However, traditional machine learning requires domain knowledge to delineate the brain regions first, followed by feature extraction from the regions. Compared with this semi-automated approach, recently developed deep learning methods have advantages since they do not require such prior knowledge; instead, deep learning methods can automatically find features that differentiate MRIs from different cohorts. In the present study, we developed a deep learning-based classification pipeline distinguishing brain MRIs of individuals with one of three types of headaches [migraine (n = 95), acute post-traumatic headache (n = 48) and persistent post-traumatic headache (n = 49)] from those of healthy controls (n = 532) and identified the brain regions that most contributed to each classification task. Our pipeline included: (i) data preprocessing; (ii) binary classification of healthy controls versus headache type using a 3D ResNet-18; and (iii) biomarker extraction from the trained 3D ResNet-18. During the classification at the second step of our pipeline, we resolved two common issues in deep learning methods, limited training data and imbalanced samples from different categories, by incorporating a large public data set and resampling among the headache cohorts. Our method achieved the following classification accuracies when tested on independent test sets: (i) migraine versus healthy controls-75% accuracy, 66.7% sensitivity and 83.3% specificity; (2) acute post-traumatic headache versus healthy controls-75% accuracy, 66.7% sensitivity and 83.3% specificity; and (3) persistent post-traumatic headache versus healthy controls-91.7% accuracy, 100% sensitivity and 83.3% specificity. The most significant biomarkers identified by the classifier for migraine were caudate, caudal anterior cingulate, superior frontal, thalamus and ventral diencephalon. For acute post-traumatic headache, lateral occipital, cuneus, lingual, pericalcarine and superior parietal regions were identified as most significant biomarkers. Finally, for persistent post-traumatic headache, the most significant biomarkers were cerebellum, middle temporal, inferior temporal, inferior parietal and superior parietal. In conclusion, our study shows that the deep learning methods can automatically detect aberrations in the brain regions associated with different headache types. It does not require any human knowledge as input which significantly reduces human effort. It uncovers the great potential of deep learning methods for classification and automatic extraction of brain imaging-based biomarkers for these headache types.

Skin lesion classification using CNNs with grouping of multi-scale attention and class-specific loss weighting

As one of the most common cancers globally, the incidence of skin cancer has been rising. Dermoscopy-based classification has become the most effective method for the diagnosis of skin lesion types due to its accuracy and non-invasive characteristics, which plays a significant role in reducing mortality. Although a great breakthrough of the task of skin lesion classification has been made with the application of convolutional neural network, the inter-class similarity and intra-class variation in skin lesions images, the high class imbalance of the dataset and the lack of ability to focus on the lesion area all affect the classification results of the model. In order to solve these problems, on the one hand, we use the grouping of multi-scale attention blocks (GMAB) to extract multi-scale fine-grained features so as to improve the model's ability to focus on the lesion area. On the other hand, we adopt the method of class-specific loss weighting for the problem of category imbalance. In this paper, we propose a deep convolution neural network dermatoscopic image classification method based on the grouping of multi-scale attention blocks and class-specific loss weighting. We evaluated our model on the HAM10000 dataset, and the results showed that the ACC and AUC of the proposed method were 91.6% and 97.1% respectively, which can achieve good results in dermatoscopic classification tasks.

MTU: A multi-tasking U-net with hybrid convolutional learning and attention modules for cancer classification and gland Segmentation in Colon Histopathological Images

A clinically comparable multi-tasking computerized deep U-Net-based model is demonstrated in this paper. It intends to offer clinical gland morphometric information and cancer grade classification to be provided as referential opinions for pathologists in order to abate human errors. It embraces enhanced feature learning capability that aids in extraction of potent multi-scale features; efficacious semantic gap recovery during feature concatenation; and successful interception of resolution-degradation and vanishing gradient problems while performing moderate computations. It is proposed by integrating three unique novel structural components namely Hybrid Convolutional Learning Units in the encoder and decoder, Attention Learning Units in skip connection, and Multi-Scalar Dilated Transitional Unit as the transitional layer in the traditional U-Net architecture. These units are composed of the amalgamated phenomenon of multi-level convolutional learning through conventional, atrous, residual, depth-wise, and point-wise convolutions which are further incorporated with target-specific attention learning and enlarged effectual receptive field size. Also, pre-processing techniques of patch-sampling, augmentation (color and morphological), stain-normalization, etc. are employed to burgeon its generalizability. To build network invariance towards digital variability, exhaustive experiments are conducted using three public datasets (Colorectal Adenocarcinoma Gland (CRAG), Gland Segmentation (GlaS) challenge, and Lung Colon-25000 (LC-25K) dataset)) and then its robustness is verified using an in-house private dataset of Hospital Colon (HosC). For the cancer classification, the proposed model achieved results of Accuracy (CRAG(95%), GlaS(97.5%), LC-25K(99.97%), HosC(99.45%)), Precision (CRAG(0.9678), GlaS(0.9768), LC-25K(1), HosC(1)), F1-score (CRAG(0.968), GlaS(0.977), LC 25K(0.9997), HosC(0.9965)), and Recall (CRAG(0.9677), GlaS(0.9767), LC-25K(0.9994), HosC(0.9931)). For the gland detection and segmentation, the proposed model achieved competitive results of F1-score (CRAG(0.924), GlaS(Test A(0.949), Test B(0.918)), LC-25K(0.916), HosC(0.959)); Object-Dice Index (CRAG(0.959), GlaS(Test A(0.956), Test B(0.909)), LC-25K(0.929), HosC(0.922)), and Object-Hausdorff Distance (CRAG(90.47), GlaS(Test A(23.17), Test B(71.53)), LC-25K(96.28), HosC(85.45)). In addition, the activation mappings for testing the interpretability of the classification decision-making process are reported by utilizing techniques of Local Interpretable Model-Agnostic Explanations, Occlusion Sensitivity, and Gradient-Weighted Class Activation Mappings. This is done to provide further evidence about the model's self-learning capability of the comparable patterns considered relevant by pathologists without any pre-requisite for annotations. These activation mapping visualization outcomes are evaluated by proficient pathologists, and they delivered these images with a class-path validation score of (CRAG(9.31), GlaS(9.25), LC-25K(9.05), and HosC(9.85)). Furthermore, the seg-path validation score of (GlaS (Test A(9.40), Test B(9.25)), CRAG(9.27), LC-25K(9.01), HosC(9.19)) given by multiple pathologists is included for the final segmented outcomes to substantiate the clinical relevance and suitability for facilitation at the clinical level. The proposed model will aid pathologists to formulate an accurate diagnosis by providing a referential opinion during the morphology assessment of histopathology images. It will reduce unintentional human error in cancer diagnosis and consequently will enhance patient survival rate.

Dermoscopy lesion classification based on GANs and a fuzzy rank-based ensemble of CNN models

Background and Objective. Skin lesion classification by using deep learning technologies is still a considerable challenge due to high similarity among classes and large intraclass differences, serious class imbalance in data, and poor classification accuracy with low robustness. Approach. To address these issues, a two-stage framework for dermoscopy lesion classification using adversarial training and a fuzzy rank-based ensemble of multilayer feature fusion convolutional neural network (CNN) models is proposed. In the first stage, dermoscopy dataset augmentation based on generative adversarial networks is proposed to obtain realistic dermoscopy lesion images, enabling significant improvement for balancing the number of lesions in each class. In the second stage, a fuzzy rank-based ensemble of multilayer feature fusion CNN models is proposed to classify skin lesions. In addition, an efficient channel integrating spatial attention module, in which a novel dilated pyramid pooling structure is designed to extract multiscale features from an enlarged receptive field and filter meaningful information of the initial features. Combining the cross-entropy loss function with the focal loss function, a novel united loss function is designed to reduce the intraclass sample distance and to focus on difficult and error-prone samples to improve the recognition accuracy of the proposed model. Main results. In this paper, the common dataset (HAM10000) is selected to conduct simulation experiments to evaluate and verify the effectiveness of the proposed method. The subjective and objective experimental results demonstrate that the proposed method is superior over the state-of-the-art methods for skin lesion classification due to its higher accuracy, specificity and robustness. Significance. The proposed method effectively improves the classification performance of the model for skin diseases, which will help doctors make accurate and efficient diagnoses, reduce the incidence rate and improve the survival rates of patients.

An xception model based on residual attention mechanism for the classification of benign and malignant gastric ulcers

To explore the application value of convolutional neural network combined with residual attention mechanism and Xception model for automatic classification of benign and malignant gastric ulcer lesions in common digestive endoscopy images under the condition of insufficient data. For the problems of uneven illumination and low resolution of endoscopic images, the original image is preprocessed by Sobel operator, etc. The algorithm model is implemented by Pytorch, and the preprocessed image is used as input data. The model is based on convolutional neural network for automatic classification and diagnosis of benign and malignant gastric ulcer lesions in small number of digestive endoscopy images. The accuracy, F1 score, sensitivity, specificity and precision of the Xception model improved by the residual attention module for the diagnosis of benign and malignant gastric ulcer lesions were 81.411%, 81.815%, 83.751%, 76.827% and 80.111%, respectively. The superposition of residual attention modules can effectively improve the feature learning ability of the model. The pretreatment of digestive endoscopy can remove the interference information on the digestive endoscopic image data extracted from the database, which is beneficial to the training of the model. The residual attention mechanism can effectively improve the classification effect of Xception convolutional neural network on benign and malignant lesions of gastric ulcer on common digestive endoscopic images.

N-Net: A novel dense fully convolutional neural network for thyroid nodule segmentation

Medical image segmentation is an essential component of computer-aided diagnosis (CAD) systems. Thyroid nodule segmentation using ultrasound images is a necessary step for the early diagnosis of thyroid diseases. An encoder-decoder based deep convolutional neural network (DCNN), like U-Net architecture and its variants, has been extensively used to deal with medical image segmentation tasks. In this article, we propose a novel N-shape dense fully convolutional neural network for medical image segmentation, referred to as N-Net. The proposed framework is composed of three major components: a multi-scale input layer, an attention guidance module, and an innovative stackable dilated convolution (SDC) block. First, we apply the multi-scale input layer to construct an image pyramid, which achieves multi-level receiver field sizes and obtains rich feature representation. After that, the U-shape convolutional network is employed as the backbone structure. Moreover, we use the attention guidance module to filter the features before several skip connections, which can transfer structural information from previous feature maps to the following layers. This module can also remove noise and reduce the negative impact of the background. Finally, we propose a stackable dilated convolution (SDC) block, which is able to capture deep semantic features that may be lost in bilinear upsampling. We have evaluated the proposed N-Net framework on a thyroid nodule ultrasound image dataset (called the TNUI-2021 dataset) and the DDTI publicly available dataset. The experimental results show that our N-Net model outperforms several state-of-the-art methods in the thyroid nodule segmentation tasks.

Multi-Class Skin Lesion Classification Using a Lightweight Dynamic Kernel Deep-Learning-Based Convolutional Neural Network

Skin is the primary protective layer of the internal organs of the body. Nowadays, due to increasing pollution and multiple other factors, various types of skin diseases are growing globally. With variable shapes and multiple types, the classification of skin lesions is a challenging task. Motivated by this spreading deformity in society, a lightweight and efficient model is proposed for the highly accurate classification of skin lesions. Dynamic-sized kernels are used in layers to obtain the best results, resulting in very few trainable parameters. Further, both ReLU and leakyReLU activation functions are purposefully used in the proposed model. The model accurately classified all of the classes of the HAM10000 dataset. The model achieved an overall accuracy of 97.85%, which is much better than multiple state-of-the-art heavy models. Further, our work is compared with some popular state-of-the-art and recent existing models.

Cancer-Net SCa: tailored deep neural network designs for detection of skin cancer from dermoscopy images

Background

Skin cancer continues to be the most frequently diagnosed form of cancer in the U.S., with not only significant effects on health and well-being but also significant economic costs associated with treatment. A crucial step to the treatment and management of skin cancer is effective early detection with key screening approaches such as dermoscopy examinations, leading to stronger recovery prognoses. Motivated by the advances of deep learning and inspired by the open source initiatives in the research community, in this study we introduce Cancer-Net SCa, a suite of deep neural network designs tailored for the detection of skin cancer from dermoscopy images that is open source and available to the general public. To the best of the authors’ knowledge, Cancer-Net SCa comprises the first machine-driven design of deep neural network architectures tailored specifically for skin cancer detection, one of which leverages attention condensers for an efficient self-attention design.

Results

We investigate and audit the behaviour of Cancer-Net SCa in a responsible and transparent manner through explainability-driven performance validation. All the proposed designs achieved improved accuracy when compared to the ResNet-50 architecture while also achieving significantly reduced architectural and computational complexity. In addition, when evaluating the decision making process of the networks, it can be seen that diagnostically relevant critical factors are leveraged rather than irrelevant visual indicators and imaging artifacts.

Conclusion

The proposed Cancer-Net SCa designs achieve strong skin cancer detection performance on the International Skin Imaging Collaboration (ISIC) dataset, while providing a strong balance between computation and architectural efficiency and accuracy. While Cancer-Net SCa is not a production-ready screening solution, the hope is that the release of Cancer-Net SCa in open source, open access form will encourage researchers, clinicians, and citizen data scientists alike to leverage and build upon them.

Glioma survival prediction from whole-brain MRI without tumor segmentation using deep attention network: a multicenter study

OBJECTIVES

To develop and validate a deep learning model for predicting overall survival from whole-brain MRI without tumor segmentation in patients with diffuse gliomas.

METHODS

In this multicenter retrospective study, two deep learning models were built for survival prediction from MRI, including a DeepRisk model built from whole-brain MRI, and an original ResNet model built from expert-segmented tumor images. Both models were developed using a training dataset (n = 935) and an internal tuning dataset (n = 156) and tested on two external test datasets (n = 194 and 150) and a TCIA dataset (n = 121). C-index, integrated Brier score (IBS), prediction error curves, and calibration curves were used to assess the model performance.

RESULTS

In total, 1556 patients were enrolled (age, 49.0 ± 13.1 years; 830 male). The DeepRisk score was an independent predictor and can stratify patients in each test dataset into three risk subgroups. The IBS and C-index for DeepRisk were 0.14 and 0.83 in external test dataset 1, 0.15 and 0.80 in external dataset 2, and 0.16 and 0.77 in TCIA dataset, respectively, which were comparable with those for original ResNet. The AUCs at 6, 12, 24, 26, and 48 months for DeepRisk ranged between 0.77 and 0.94. Combining DeepRisk score with clinicomolecular factors resulted in a nomogram with a better calibration and classification accuracy (net reclassification improvement 0.69, p < 0.001) than the clinical nomogram.

CONCLUSIONS

DeepRisk that obviated the need of tumor segmentation can predict glioma survival from whole-brain MRI and offers incremental prognostic value.

KEY POINTS

• DeepRisk can predict overall survival directly from whole-brain MRI without tumor segmentation. • DeepRisk achieves comparable accuracy in survival prediction with deep learning model built using expert-segmented tumor images. • DeepRisk has independent and incremental prognostic value over existing clinical parameters and IDH mutation status.

Multi-Task Weakly-Supervised Attention Network for Dementia Status Estimation With Structural MRI

Accurate prediction of clinical scores (of neuropsychological tests) based on noninvasive structural magnetic resonance imaging (MRI) helps understand the pathological stage of dementia (e.g., Alzheimer's disease (AD)) and forecast its progression. Existing machine/deep learning approaches typically preselect dementia-sensitive brain locations for MRI feature extraction and model construction, potentially leading to undesired heterogeneity between different stages and degraded prediction performance. Besides, these methods usually rely on prior anatomical knowledge (e.g., brain atlas) and time-consuming nonlinear registration for the preselection of brain locations, thereby ignoring individual-specific structural changes during dementia progression because all subjects share the same preselected brain regions. In this article, we propose a multi-task weakly-supervised attention network (MWAN) for the joint regression of multiple clinical scores from baseline MRI scans. Three sequential components are included in MWAN: 1) a backbone fully convolutional network for extracting MRI features; 2) a weakly supervised dementia attention block for automatically identifying subject-specific discriminative brain locations; and 3) an attention-aware multitask regression block for jointly predicting multiple clinical scores. The proposed MWAN is an end-to-end and fully trainable deep learning model in which dementia-aware holistic feature learning and multitask regression model construction are integrated into a unified framework. Our MWAN method was evaluated on two public AD data sets for estimating clinical scores of mini-mental state examination (MMSE), clinical dementia rating sum of boxes (CDRSB), and AD assessment scale cognitive subscale (ADAS-Cog). Quantitative experimental results demonstrate that our method produces superior regression performance compared with state-of-the-art methods. Importantly, qualitative results indicate that the dementia-sensitive brain locations automatically identified by our MWAN method well retain individual specificities and are biologically meaningful.

SCDNet: A Deep Learning-Based Framework for the Multiclassification of Skin Cancer Using Dermoscopy Images

Skin cancer is a deadly disease, and its early diagnosis enhances the chances of survival. Deep learning algorithms for skin cancer detection have become popular in recent years. A novel framework based on deep learning is proposed in this study for the multiclassification of skin cancer types such as Melanoma, Melanocytic Nevi, Basal Cell Carcinoma and Benign Keratosis. The proposed model is named as SCDNet which combines Vgg16 with convolutional neural networks (CNN) for the classification of different types of skin cancer. Moreover, the accuracy of the proposed method is also compared with the four state-of-the-art pre-trained classifiers in the medical domain named Resnet 50, Inception v3, AlexNet and Vgg19. The performance of the proposed SCDNet classifier, as well as the four state-of-the-art classifiers, is evaluated using the ISIC 2019 dataset. The accuracy rate of the proposed SDCNet is 96.91% for the multiclassification of skin cancer whereas, the accuracy rates for Resnet 50, Alexnet, Vgg19 and Inception-v3 are 95.21%, 93.14%, 94.25% and 92.54%, respectively. The results showed that the proposed SCDNet performed better than the competing classifiers.

Adversarial multimodal fusion with attention mechanism for skin lesion classification using clinical and dermoscopic images

Accurate skin lesion diagnosis requires a great effort from experts to identify the characteristics from clinical and dermoscopic images. Deep multimodal learning-based methods can reduce intra- and inter-reader variability and improve diagnostic accuracy compared to the single modality-based methods. This study develops a novel method, named adversarial multimodal fusion with attention mechanism (AMFAM), to perform multimodal skin lesion classification. Specifically, we adopt a discriminator that uses adversarial learning to enforce the feature extractor to learn the correlated information explicitly. Moreover, we design an attention-based reconstruction strategy to encourage the feature extractor to concentrate on learning the features of the lesion area, thus, enhancing the feature vector from each modality with more discriminative information. Unlike existing multimodal-based approaches, which only focus on learning complementary features from dermoscopic and clinical images, our method considers both correlated and complementary information of the two modalities for multimodal fusion. To verify the effectiveness of our method, we conduct comprehensive experiments on a publicly available multimodal and multi-task skin lesion classification dataset: 7-point criteria evaluation database. The experimental results demonstrate that our proposed method outperforms the current state-of-the-art methods and improves the average AUC score by above 2% on the test set.

Semi-Supervised Medical Image Classification Based on Attention and Intrinsic Features of Samples

The training of deep neural networks usually requires a lot of high-quality data with good annotations to obtain good performance. However, in clinical medicine, obtaining high-quality marker data is laborious and expensive because it requires the professional skill of clinicians. In this paper, based on the consistency strategy, we propose a new semi-supervised model for medical image classification which introduces a self-attention mechanism into the backbone network to learn more meaningful features in image classification tasks and uses the improved version of focal loss at the supervision loss to reduce the misclassification of samples. Finally, we add a consistency loss similar to the unsupervised consistency loss to encourage the model to learn more about the internal features of unlabeled samples. Our method achieved 94.02% AUC and 72.03% Sensitivity on the ISIC 2018 dataset and 79.74% AUC on the ChestX-ray14 dataset. These results show the effectiveness of our method in single-label and multi-label classification.

Fusion of Local and Global Feature Representation With Sparse Autoencoder for Improved Melanoma Classification

Automated skin cancer diagnosis is challenging due to inter-class uniformity, intra-class variation, and the complex structure of dermoscopy images. Convolutional neural networks (CNN) have recently made considerable progress in melanoma classification, even in the presence of limited skin images. One of the drawbacks of these methods is the loss of image details caused by downsampling high-resolution skin images to a low resolution. Further, most approaches extract features only from the whole skin image. This paper proposes an ensemble feature fusion and sparse autoencoder (SAE) based framework to overcome the above issues and improve melanoma classification performance. The proposed method extracts features from two streams, local and global, using a pre-trained CNN model. The local stream extracts features from image patches, while the global stream derives features from the whole skin image, preserving both local and global representation. The features are then fused, and an SAE framework is subsequently designed to enrich the feature representation further. The proposed method is validated on ISIC 2016 dataset and the experimental results indicate the superiority of the proposed approach.

An efficient U-shaped network combined with edge attention module and context pyramid fusion for skin lesion segmentation

Skin lesion segmentation is an important process in skin diagnosis, but still a challenging problem due to the variety of shapes, colours, and boundaries of melanoma. In this paper, we propose a novel and efficient U-shaped network named EAM-CPFNet, which combines with edge attention module (EAM) and context pyramid fusion (CPF) to improve the performance of the skin lesion segmentation. First, we design a plug-and-play module named edge attention module (EAM), which is used to highlight the edge information learned in the encoder. Secondly, we integrate two pyramid modules collectively named context pyramid fusion (CPF) for context information fusion. One is multiple global pyramid guidance (GPG) modules, which replace the skip connections between the encoder and the decoder to capture global context information, and the other is scale-aware pyramid fusion (SAPF) module, which is designed to dynamically fuse multi-scale context information in high-level features by utilizing spatial and channel attention mechanisms. Furthermore, we introduce full-scale skip connections to enhance different levels of global context information. We evaluate the proposed method on the publicly available ISIC2018 dataset, and the experimental results demonstrate that our proposed method is very competitive compared with other state-of-the-art methods for the skin lesion segmentation.

Deep Clustering via Center-Oriented Margin Free-Triplet Loss for Skin Lesion Detection in Highly Imbalanced Datasets

Melanoma is a fatal skin cancer that is curable and has dramatically increasing survival rate when diagnosed at early stages. Learning-based methods hold significant promise for the detection of melanoma from dermoscopic images. However, since melanoma is a rare disease, existing databases of skin lesions predominantly contain highly imbalanced numbers of benign versus malignant samples. In turn, this imbalance introduces substantial bias in classification models due to the statistical dominance of the majority class. To address this issue, we introduce a deep clustering approach based on the latent-space embedding of dermoscopic images. Clustering is achieved using a novel center-oriented margin-free triplet loss (COM-Triplet) enforced on image embeddings from a convolutional neural network backbone. The proposed method aims to form maximally-separated cluster centers as opposed to minimizing classification error, so it is less sensitive to class imbalance. To avoid the need for labeled data, we further propose to implement COM-Triplet based on pseudo-labels generated by a Gaussian mixture model (GMM). Comprehensive experiments show that deep clustering with COM-Triplet loss outperforms clustering with triplet loss, and competing classifiers in both supervised and unsupervised settings.

Attention-based Dynamic Subspace Learners for Medical Image Analysis

Learning similarity is a key aspect in medical image analysis, particularly in recommendation systems or in uncovering the interpretation of anatomical data in images. Most existing methods learn such similarities in the embedding space over image sets using a single metric learner. Images, however, have a variety of object attributes such as color, shape, or artifacts. Encoding such attributes using a single metric learner is inadequate and may fail to generalize. Instead, multiple learners could focus on separate aspects of these attributes in subspaces of an overarching embedding. This, however, implies the number of learners to be found empirically for each new dataset. This work, Dynamic Subspace Learners, proposes to dynamically exploit multiple learners by removing the need of knowing apriori the number of learners and aggregating new subspace learners during training. Furthermore, the visual interpretability of such subspace learning is enforced by integrating an attention module into our method. This integrated attention mechanism provides a visual insight of discriminative image features that contribute to the clustering of image sets and a visual explanation of the embedding features. The benefits of our attention-based dynamic subspace learners are evaluated in the application of image clustering, image retrieval, and weakly supervised segmentation. Our method achieves competitive results with the performances of multiple learners baselines and significantly outperforms the classification network in terms of clustering and retrieval scores on three different public benchmark datasets. Moreover, our method also provides an attention map generated directly during inference to illustrate the visual interpretability of the embedding features. These attention maps offer a proxy-labels, which improves the segmentation accuracy up to 15% in Dice scores when compared to state-of-the-art interpretation techniques.

Multi-branch fusion auxiliary learning for the detection of pneumonia from chest X-ray images

Lung infections caused by bacteria and viruses are infectious and require timely screening and isolation, and different types of pneumonia require different treatment plans. Therefore, finding a rapid and accurate screening method for lung infections is critical. To achieve this goal, we proposed a multi-branch fusion auxiliary learning (MBFAL) method for pneumonia detection from chest X-ray (CXR) images. The MBFAL method was used to perform two tasks through a double-branch network. The first task was to recognize the absence of pneumonia (normal), COVID-19, other viral pneumonia and bacterial pneumonia from CXR images, and the second task was to recognize the three types of pneumonia from CXR images. The latter task was used to assist the learning of the former task to achieve a better recognition effect. In the process of auxiliary parameter updating, the feature maps of different branches were fused after sample screening through label information to enhance the model’s ability to recognize case of pneumonia without impacting its ability to recognize normal cases. Experiments show that an average classification accuracy of 95.61% is achieved using MBFAL. The single class accuracy for normal, COVID-19, other viral pneumonia and bacterial pneumonia was 98.70%, 99.10%, 96.60% and 96.80%, respectively, and the recall was 97.20%, 98.60%, 96.10% and 89.20%, respectively, using the MBFAL method. Compared with the baseline model and the model constructed using the above methods separately, better results for the rapid screening of pneumonia were achieved using MBFAL.

Dermoscopic Image Classification Method Using an Ensemble of Fine-Tuned Convolutional Neural Networks

Aiming at the problems of large intra-class differences, small inter-class differences, low contrast, and small and unbalanced datasets in dermoscopic images, this paper proposes a dermoscopic image classification method based on an ensemble of fine-tuned convolutional neural networks. By reconstructing the fully connected layers of the three pretrained models of Xception, ResNet50, and Vgg-16 and then performing transfer learning and fine-tuning the three pretrained models with the ISIC 2016 Challenge official skin dataset, we integrated the outputs of the three base models using a weighted fusion ensemble strategy in order to obtain a final prediction result able to distinguish whether a dermoscopic image indicates malignancy. The experimental results show that the accuracy of the ensemble model is 86.91%, the precision is 85.67%, the recall is 84.03%, and the F1-score is 84.84%, with these four evaluation metrics being better than those of the three basic models and better than some classical methods, proving the effectiveness and feasibility of the proposed method.

An Effective Skin Cancer Classification Mechanism via Medical Vision Transformer

Skin Cancer (SC) is considered the deadliest disease in the world, killing thousands of people every year. Early SC detection can increase the survival rate for patients up to 70%, hence it is highly recommended that regular head-to-toe skin examinations are conducted to determine whether there are any signs or symptoms of SC. The use of Machine Learning (ML)-based methods is having a significant impact on the classification and detection of SC diseases. However, there are certain challenges associated with the accurate classification of these diseases such as a lower detection accuracy, poor generalization of the models, and an insufficient amount of labeled data for training. To address these challenges, in this work we developed a two-tier framework for the accurate classification of SC. During the first stage of the framework, we applied different methods for data augmentation to increase the number of image samples for effective training. As part of the second tier of the framework, taking into consideration the promising performance of the Medical Vision Transformer (MVT) in the analysis of medical images, we developed an MVT-based classification model for SC. This MVT splits the input image into image patches and then feeds these patches to the transformer in a sequence structure, like word embedding. Finally, Multi-Layer Perceptron (MLP) is used to classify the input image into the corresponding class. Based on the experimental results achieved on the Human Against Machine (HAM10000) datasets, we concluded that the proposed MVT-based model achieves better results than current state-of-the-art techniques for SC classification.

Automatic lung segmentation in chest X-ray images using improved U-Net

The automatic segmentation of the lung region for chest X-ray (CXR) can help doctors diagnose many lung diseases. However, extreme lung shape changes and fuzzy lung regions caused by serious lung diseases may incorrectly make the automatic lung segmentation model. We improved the U-Net network by using the pre-training Efficientnet-b4 as the encoder and the Residual block and the LeakyReLU activation function in the decoder. The network can extract Lung field features efficiently and avoid the gradient instability caused by the multiplication effect in gradient backpropagation. Compared with the traditional U-Net model, our method improves about 2.5% dice coefficient and 6% Jaccard Index for the two benchmark lung segmentation datasets. Our model improves about 5% dice coefficient and 9% Jaccard Index for the private lung segmentation datasets compared with the traditional U-Net model. Comparative experiments show that our method can improve the accuracy of lung segmentation of CXR images and it has a lower standard deviation and good robustness.

A Multi-Scale Densely Connected Convolutional Neural Network for Automated Thyroid Nodule Classification

Automated thyroid nodule classification in ultrasound images is an important way to detect thyroid nodules and to make a more accurate diagnosis. In this paper, we propose a novel deep convolutional neural network (CNN) model, called n-ClsNet, for thyroid nodule classification. Our model consists of a multi-scale classification layer, multiple skip blocks, and a hybrid atrous convolution (HAC) block. The multi-scale classification layer first obtains multi-scale feature maps in order to make full use of image features. After that, each skip-block propagates information at different scales to learn multi-scale features for image classification. Finally, the HAC block is used to replace the downpooling layer so that the spatial information can be fully learned. We have evaluated our n-ClsNet model on the TNUI-2021 dataset. The proposed n-ClsNet achieves an average accuracy (ACC) score of 93.8% in the thyroid nodule classification task, which outperforms several representative state-of-the-art classification methods.

Effective high-to-low-level feature aggregation network for endoscopic image classification

PURPOSE

The accuracy improvement in endoscopic image classification matters to the endoscopists in diagnosing and choosing suitable treatment for patients. Existing CNN-based methods for endoscopic image classification tend to use the deepest abstract features without considering the contribution of low-level features, while the latter is of great significance in the actual diagnosis of intestinal diseases.

METHODS

To make full use of both high-level and low-level features, we propose a novel two-stream network for endoscopic image classification. Specifically, the backbone stream is utilized to extract high-level features. In the fusion stream, low-level features are generated by a bottom-up multi-scale gradual integration (BMGI) method, and the input of BMGI is refined by top-down attention learning modules. Besides, a novel correction loss is proposed to clarify the relationship between high-level and low-level features.

RESULTS

Experiments on the KVASIR dataset demonstrate that the proposed framework can obtain an overall classification accuracy of 97.33% with Kappa coefficient of 95.25%. Compared to the existing models, the two evaluation indicators have increased by 2% and 2.25%, respectively, at least.

CONCLUSION

In this study, we proposed a two-stream network that fuses the high-level and low-level features for endoscopic image classification. The experiment results show that the high-to-low-level feature can better represent the endoscopic image and enable our model to outperform several state-of-the-art classification approaches. In addition, the proposed correction loss could regularize the consistency between backbone stream and fusion stream. Thus, the fused feature can reduce the intra-class distances and make accurate label prediction.

Characterization of motion patterns by a spatio-temporal saliency descriptor in cardiac cine MRI

BACKGROUND AND OBJECTIVE

Abnormalities of the heart motion reveal the presence of a disease. However, a quantitative interpretation of the motion is still a challenge due to the complex dynamics of the heart. This work proposes a quantitative characterization of regional cardiac motion patterns in cine magnetic resonance imaging (MRI) by a novel spatio-temporal saliency descriptor.

METHOD

The strategy starts by dividing the cardiac sequence into a progression of scales which are in due turn mapped to a feature space of regional orientation changes, mimicking the multi-resolution decomposition of oriented primitive changes of visual systems. These changes are estimated as the difference between a particular time and the rest of the sequence. This decomposition is then temporarily and regionally integrated for a particular orientation and then for the set of different orientations. A final spatio-temporal 4D saliency map is obtained as the summation of the previously integrated information for the available scales. The saliency dispersion of this map was computed in standard cardiac locations as a measure of the regional motion pattern and was applied to discriminate control and hypertrophic cardiomyopathy (HCM) subjects during the diastolic phase.

RESULTS

Salient motion patterns were estimated from an experimental set, which consisted of 3D sequences acquired by MRI from 108 subjects (33 control, 35 HCM, 20 dilated cardiomyopathy (DCM), and 20 myocardial infarction (MINF) from heterogeneous datasets). HCM and control subjects were classified by an SVM that learned the salient motion patterns estimated from the presented strategy, by achieving a 94% AUC. In addition, statistical differences (test t-student, p<0.05) were found among groups of disease in the septal and anterior ventricular segments at both the ED and ES, with salient motion characteristics aligned with existing knowledge on the diseases.

CONCLUSIONS

Regional wall motion abnormality in the apical, anterior, basal, and inferior segments was associated with the saliency dispersion in HCM, DCM, and MINF compared to healthy controls during the systolic and diastolic phases. This saliency analysis may be used to detect subtle changes in heart function.

Single Model Deep Learning on Imbalanced Small Datasets for Skin Lesion Classification

Deep convolutional neural network (DCNN) models have been widely explored for skin disease diagnosis and some of them have achieved the diagnostic outcomes comparable or even superior to those of dermatologists. However, broad implementation of DCNN in skin disease detection is hindered by small size and data imbalance of the publically accessible skin lesion datasets. This paper proposes a novel single-model based strategy for classification of skin lesions on small and imbalanced datasets. First, various DCNNs are trained on different small and imbalanced datasets to verify that the models with moderate complexity outperform the larger models. Second, regularization DropOut and DropBlock are added to reduce overfitting and a Modified RandAugment augmentation strategy is proposed to deal with the defects of sample underrepresentation in the small dataset. Finally, a novel Multi-Weighted New Loss (MWNL) function and an end-to-end cumulative learning strategy (CLS) are introduced to overcome the challenge of uneven sample size and classification difficulty and to reduce the impact of abnormal samples on training. By combining Modified RandAugment, MWNL and CLS, our single DCNN model method achieved the classification accuracy comparable or superior to those of multiple ensembling models on different dermoscopic image datasets. Our study shows that this method is able to achieve a high classification performance at a low cost of computational resources and inference time, potentially suitable to implement in mobile devices for automated screening of skin lesions and many other malignancies in low resource settings.

A Low-Cost High-Performance Data Augmentation for Deep Learning-Based Skin Lesion Classification

Objective and Impact Statement. There is a need to develop high-performance and low-cost data augmentation strategies for intelligent skin cancer screening devices that can be deployed in rural or underdeveloped communities. The proposed strategy can not only improve the classification performance of skin lesions but also highlight the potential regions of interest for clinicians' attention. This strategy can also be implemented in a broad range of clinical disciplines for early screening and automatic diagnosis of many other diseases in low resource settings. Methods. We propose a high-performance data augmentation strategy of search space 101, which can be combined with any model through a plug-and-play mode and search for the best argumentation method for a medical database with low resource cost. Results. With EfficientNets as a baseline, the best BACC of HAM10000 is 0.853, outperforming the other published models of "single-model and no-external-database" for ISIC 2018 Lesion Diagnosis Challenge (Task 3). The best average AUC performance on ISIC 2017 achieves 0.909 (±0.015), exceeding most of the ensembling models and those using external datasets. Performance on Derm7pt archives the best BACC of 0.735 (±0.018) ahead of all other related studies. Moreover, the model-based heatmaps generated by Grad-CAM++ verify the accurate selection of lesion features in model judgment, further proving the scientific rationality of model-based diagnosis. Conclusion. The proposed data augmentation strategy greatly reduces the computational cost for clinically intelligent diagnosis of skin lesions. It may also facilitate further research in low-cost, portable, and AI-based mobile devices for skin cancer screening and therapeutic guidance.