Label Co-Occurrence Learning With Graph Convolutional Networks for Multi-Label Chest X-Ray Image Classification

CorLabelNet: a comprehensive framework for multi-label chest X-ray image classification with correlation guided discriminant feature learning and oversampling

Recent advancements in deep learning techniques have significantly improved multi-label chest X-ray (CXR) image classification for clinical diagnosis. However, most previous studies neither effectively learn label correlations nor take full advantage of them to improve multi-label classification performance. In addition, different labels of CXR images are usually severely imbalanced, resulting in the model exhibiting a bias towards the majority class. To address these challenges, we introduce a framework that not only learns label correlations but also utilizes them to guide the learning of features and the process of oversampling. In this paper, our approach incorporates self-attention to capture high-order label correlations and considers label correlations from both global and local perspectives. Then, we propose a consistency constraint and a multi-label contrastive loss to enhance feature learning. To alleviate the imbalance issue, we further propose an oversampling approach that exploits the learned label correlation to identify crucial seed samples for oversampling. Our approach repeats 5-fold cross-validation process experiments three times and achieves the best performance on both the CheXpert and ChestX-Ray14 datasets. Learning accurate label correlation is significant for multi-label classification and taking full advantage of label correlations is beneficial for discriminative feature learning and oversampling. A comparative analysis with the state-of-the-art approaches highlights the effectiveness of our proposed methods.

Label-Aware Dual Graph Neural Networks for Multi-Label Fundus Image Classification

Fundus disease is a complex and universal disease involving a variety of pathologies. Its early diagnosis using fundus images can effectively prevent further diseases and provide targeted treatment plans for patients. Recent deep learning models for classification of this disease are gradually emerging as a critical research field, which is attracting widespread attention. However, in practice, most of the existing methods only focus on local visual cues of a single image, and ignore the underlying explicit interaction similarity between subjects and correlation information among pathologies in fundus diseases. In this paper, we propose a novel label-aware dual graph neural networks for multi-label fundus image classification that consists of population-based graph representation learning and pathology-based graph representation learning modules. Specifically, we first construct a population-based graph by integrating image features and non-image information to learn patient's representations by incorporating associations between subjects. Then, we represent pathologies as a sparse graph where its nodes are associated with pathology-based feature vectors and the edges correspond to probability of the co-occurrence of labels to generate a set of classifier scores by the propagation of multi-layer graph information. Finally, our model can adaptively recalibrate multi-label outputs. Detailed experiments and analysis of our results show the effectiveness of our method compared with state-of-the-art multi-label fundus image classification methods.

A New Benchmark: Clinical Uncertainty and Severity Aware Labeled Chest X-Ray Images With Multi-Relationship Graph Learning

Chest radiography, commonly known as CXR, is frequently utilized in clinical settings to detect cardiopulmonary conditions. However, even seasoned radiologists might offer different evaluations regarding the seriousness and uncertainty associated with observed abnormalities. Previous research has attempted to utilize clinical notes to extract abnormal labels for training deep-learning models in CXR image diagnosis. However, these methods often neglected the varying degrees of severity and uncertainty linked to different labels. In our study, we initially assembled a comprehensive new dataset of CXR images based on clinical textual data, which incorporated radiologists' assessments of uncertainty and severity. Using this dataset, we introduced a multi-relationship graph learning framework that leverages spatial and semantic relationships while addressing expert uncertainty through a dedicated loss function. Our research showcases a notable enhancement in CXR image diagnosis and the interpretability of the diagnostic model, surpassing existing state-of-the-art methodologies. The dataset address of disease severity and uncertainty we extracted is: https://physionet.org/content/cad-chest/1.0/.

CLARE-XR: explainable regression-based classification of chest radiographs with label embeddings

An automatic system for pathology classification in chest X-ray scans needs more than predictive performance, since providing explanations is deemed essential for fostering end-user trust, improving decision-making, and regulatory compliance. CLARE-XR is a novel methodology that, when presented with an X-ray image, identifies the associated pathologies and provides explanations based on the presentation of similar cases. The diagnosis is achieved using a regression model that maps an image into a 2D latent space containing the reference coordinates of all findings. The references are generated once through label embedding, before the regression step, by converting the original binary ground-truth annotations to 2D coordinates. The classification is inferred minding the distance from the coordinates of an inference image to the reference coordinates. Furthermore, as the regressor is trained on a known set of images, the distance from the coordinates of an inference image to the coordinates of the training set images also allows retrieving similar instances, mimicking the common clinical practice of comparing scans to confirm diagnoses. This inherently interpretable framework discloses specific classification rules and visual explanations through automatic image retrieval methods, outperforming the multi-label ResNet50 classification baseline across multiple evaluation settings on the NIH ChestX-ray14 dataset.

Multi-Label Chest X-Ray Image Classification With Single Positive Labels

Deep learning approaches for multi-label Chest X-ray (CXR) images classification usually require large-scale datasets. However, acquiring such datasets with full annotations is costly, time-consuming, and prone to noisy labels. Therefore, we introduce a weakly supervised learning problem called Single Positive Multi-label Learning (SPML) into CXR images classification (abbreviated as SPML-CXR), in which only one positive label is annotated per image. A simple solution to SPML-CXR problem is to assume that all the unannotated pathological labels are negative, however, it might introduce false negative labels and decrease the model performance. To this end, we present a Multi-level Pseudo-label Consistency (MPC) framework for SPML-CXR. First, inspired by the pseudo-labeling and consistency regularization in semi-supervised learning, we construct a weak-to-strong consistency framework, where the model prediction on weakly-augmented image is treated as the pseudo label for supervising the model prediction on a strongly-augmented version of the same image, and define an Image-level Perturbation-based Consistency (IPC) regularization to recover the potential mislabeled positive labels. Besides, we incorporate Random Elastic Deformation (RED) as an additional strong augmentation to enhance the perturbation. Second, aiming to expand the perturbation space, we design a perturbation stream to the consistency framework at the feature-level and introduce a Feature-level Perturbation-based Consistency (FPC) regularization as a supplement. Third, we design a Transformer-based encoder module to explore the sample relationship within each mini-batch by a Batch-level Transformer-based Correlation (BTC) regularization. Extensive experiments on the CheXpert and MIMIC-CXR datasets have shown the effectiveness of our MPC framework for solving the SPML-CXR problem.

Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge

Many real-world image recognition problems, such as diagnostic medical imaging exams, are "long-tailed" - there are a few common findings followed by many more relatively rare conditions. In chest radiography, diagnosis is both a long-tailed and multi-label problem, as patients often present with multiple findings simultaneously. While researchers have begun to study the problem of long-tailed learning in medical image recognition, few have studied the interaction of label imbalance and label co-occurrence posed by long-tailed, multi-label disease classification. To engage with the research community on this emerging topic, we conducted an open challenge, CXR-LT, on long-tailed, multi-label thorax disease classification from chest X-rays (CXRs). We publicly release a large-scale benchmark dataset of over 350,000 CXRs, each labeled with at least one of 26 clinical findings following a long-tailed distribution. We synthesize common themes of top-performing solutions, providing practical recommendations for long-tailed, multi-label medical image classification. Finally, we use these insights to propose a path forward involving vision-language foundation models for few- and zero-shot disease classification.

Automated thorax disease diagnosis using multi-branch residual attention network

Chest X-ray (CXR) is an extensively utilized radiological modality for supporting the diagnosis of chest diseases. However, existing research approaches suffer from limitations in effectively integrating multi-scale CXR image features and are also hindered by imbalanced datasets. Therefore, there is a pressing need for further advancement in computer-aided diagnosis (CAD) of thoracic diseases. To tackle these challenges, we propose a multi-branch residual attention network (MBRANet) for thoracic disease diagnosis. MBRANet comprises three components. Firstly, to address the issue of inadequate extraction of spatial and positional information by the convolutional layer, a novel residual structure incorporating a coordinate attention (CA) module is proposed to extract features at multiple scales. Next, based on the concept of a Feature Pyramid Network (FPN), we perform multi-scale feature fusion in the following manner. Thirdly, we propose a novel Multi-Branch Feature Classifier (MFC) approach, which leverages the class-specific residual attention (CSRA) module for classification instead of relying solely on the fully connected layer. In addition, the designed BCEWithLabelSmoothing loss function improves the generalization ability and mitigates the problem of class imbalance by introducing a smoothing factor. We evaluated MBRANet on the ChestX-Ray14, CheXpert, MIMIC-CXR, and IU X-Ray datasets and achieved average AUCs of 0.841, 0.895, 0.805, and 0.745, respectively. Our method outperformed state-of-the-art baselines on these benchmark datasets.

Label-Aware Calibration and Relation-Preserving in Visual Intention Understanding

Visual intention understanding is a challenging task that explores the hidden intention behind the images of publishers in social media. Visual intention represents implicit semantics, whose ambiguous definition inevitably leads to label shifting and label blemish. The former indicates that the same image delivers intention discrepancies under different data augmentations, while the latter represents that the label of intention data is susceptible to errors or omissions during the annotation process. This paper proposes a novel method, called Label-aware Calibration and Relation-preserving (LabCR) to alleviate the above two problems from both intra-sample and inter-sample views. First, we disentangle the multiple intentions into a single intention for explicit distribution calibration in terms of the overall and the individual. Calibrating the class probability distributions in augmented instance pairs provides consistent inferred intention to address label shifting. Second, we utilize the intention similarity to establish correlations among samples, which offers additional supervision signals to form correlation alignments in instance pairs. This strategy alleviates the effect of label blemish. Extensive experiments have validated the superiority of the proposed method LabCR in visual intention understanding and pedestrian attribute recognition. Code is available at https://github.com/ShiQingHongYa/LabCR.

Patch-Level Feature Selection for Thoracic Disease Classification by Chest X-ray Images Using Information Bottleneck

Chest X-ray (CXR) examination serves as a widely employed clinical test in medical diagnostics. Many studied have tried to apply artificial intelligence (AI) programs to analyze CXR images. Despite numerous positive outcomes, assessing the applicability of AI models for comprehensive diagnostic support remains a formidable challenge. We observed that, even when AI models exhibit high accuracy on one dataset, their performance may deteriorate when tested on another. To address this issue, we propose incorporating a variational information bottleneck (VIB) at the patch level to enhance the generalizability of diagnostic support models. The VIB introduces a probabilistic model aimed at approximating the posterior distribution of latent variables given input data, thereby enhancing the model's generalization capabilities on unseen data. Unlike the conventional VIB approaches that flatten features and use a re-parameterization trick to sample a new latent feature, our method applies the trick to 2D feature maps. This design allows only important pixels to respond, and the model will select important patches in an image. Moreover, the proposed patch-level VIB seamlessly integrates with various convolutional neural networks, offering a versatile solution to improve performance. Experimental results illustrate enhanced accuracy in standard experiment settings. In addition, the method shows robust improvement when training and testing on different datasets.

Label correlation guided discriminative label feature learning for multi-label chest image classification

BACKGROUND AND OBJECTIVE

Multi-label Chest X-ray (CXR) images often contain rich label relationship information, which is beneficial to improve classification performance. However, because of the intricate relationships among labels, most existing works fail to effectively learn and make full use of the label correlations, resulting in limited classification performance. In this study, we propose a multi-label learning framework that learns and leverages the label correlations to improve multi-label CXR image classification.

METHODS

In this paper, we capture the global label correlations through the self-attention mechanism. Meanwhile, to better utilize label correlations for guiding feature learning, we decompose the image-level features into label-level features. Furthermore, we enhance label-level feature learning in an end-to-end manner by a consistency constraint between global and local label correlations, and a label correlation guided multi-label supervised contrastive loss.

RESULTS

To demonstrate the superior performance of our proposed approach, we conduct three times 5-fold cross-validation experiments on the CheXpert dataset. Our approach obtains an average F1 score of 44.6% and an AUC of 76.5%, achieving a 7.7% and 1.3% improvement compared to the state-of-the-art results.

CONCLUSION

More accurate label correlations and full utilization of the learned label correlations help learn more discriminative label-level features. Experimental results demonstrate that our approach achieves exceptionally competitive performance compared to the state-of-the-art algorithms.

Chest x-ray diagnosis via spatial-channel high-order attention representation learning

Objective. Chest x-ray image representation and learning is an important problem in computer-aided diagnostic area. Existing methods usually adopt CNN or Transformers for feature representation learning and focus on learning effective representations for chest x-ray images. Although good performance can be obtained, however, these works are still limited mainly due to the ignorance of mining the correlations of channels and pay little attention on the local context-aware feature representation of chest x-ray image.Approach. To address these problems, in this paper, we propose a novel spatial-channel high-order attention model (SCHA) for chest x-ray image representation and diagnosis. The proposed network architecture mainly contains three modules, i.e. CEBN, SHAM and CHAM. To be specific, firstly, we introduce a context-enhanced backbone network by employing multi-head self-attention to extract initial features for the input chest x-ray images. Then, we develop a novel SCHA which contains both spatial and channel high-order attention learning branches. For the spatial branch, we develop a novel local biased self-attention mechanism which can capture both local and long-range global dependences of positions to learn rich context-aware representation. For the channel branch, we employ Brownian Distance Covariance to encode the correlation information of channels and regard it as the image representation. Finally, the two learning branches are integrated together for the final multi-label diagnosis classification and prediction.Main results. Experiments on the commonly used datasets including ChestX-ray14 and CheXpert demonstrate that our proposed SCHA approach can obtain better performance when comparing many related approaches.Significance. This study obtains a more discriminative method for chest x-ray classification and provides a technique for computer-aided diagnosis.

CheXNet and feature pyramid network: a fusion deep learning architecture for multilabel chest X-Ray clinical diagnoses classification

The existing multilabel X-Ray image learning tasks generally contain much information on pathology co-occurrence and interdependency, which is very important for clinical diagnosis. However, the challenging part of this subject is to accurately diagnose multiple diseases that occurred in a single X-Ray image since multiple levels of features are generated in the images, and create different features as in single label detection. Various works were developed to address this challenge with proposed deep learning architectures to improve classification performance and enrich diagnosis results with multi-probability disease detection. The objective is to create an accurate result and a faster inference system to support a quick diagnosis in the medical system. To contribute to this state-of-the-art, we designed a fusion architecture, CheXNet and Feature Pyramid Network (FPN), to classify and discriminate multiple thoracic diseases from chest X-Rays. This concept enables the model to extract while creating a pyramid of feature maps with different spatial resolutions that capture low-level and high-level semantic information to encounter multiple features. The model's effectiveness is evaluated using the NIH ChestXray14 dataset, with the Area Under Curve (AUC) and accuracy metrics used to compare the results against other cutting-edge approaches. The overall results demonstrate that our method outperforms other approaches and has become promising for multilabel disease classification in chest X-Rays, with potential applications in clinical practice. The result demonstrated that we achieved an average AUC of 0.846 and an accuracy of 0.914. Further, our proposed architecture diagnoses images in 0.013 s, faster than the latest approaches.

MRChexNet: Multi-modal bridge and relational learning for thoracic disease recognition in chest X-rays

While diagnosing multiple lesion regions in chest X-ray (CXR) images, radiologists usually apply pathological relationships in medicine before making decisions. Therefore, a comprehensive analysis of labeling relationships in different data modes is essential to improve the recognition performance of the model. However, most automated CXR diagnostic methods that consider pathological relationships treat different data modalities as independent learning objects, ignoring the alignment of pathological relationships among different data modalities. In addition, some methods that use undirected graphs to model pathological relationships ignore the directed information, making it difficult to model all pathological relationships accurately. In this paper, we propose a novel multi-label CXR classification model called MRChexNet that consists of three modules: a representation learning module (RLM), a multi-modal bridge module (MBM) and a pathology graph learning module (PGL). RLM captures specific pathological features at the image level. MBM performs cross-modal alignment of pathology relationships in different data modalities. PGL models directed relationships between disease occurrences as directed graphs. Finally, the designed graph learning block in PGL performs the integrated learning of pathology relationships in different data modalities. We evaluated MRChexNet on two large-scale CXR datasets (ChestX-Ray14 and CheXpert) and achieved state-of-the-art performance. The mean area under the curve (AUC) scores for the 14 pathologies were 0.8503 (ChestX-Ray14) and 0.8649 (CheXpert). MRChexNet effectively aligns pathology relationships in different modalities and learns more detailed correlations between pathologies. It demonstrates high accuracy and generalization compared to competing approaches. MRChexNet can contribute to thoracic disease recognition in CXR.

Efficient Thorax Disease Classification and Localization Using DCNN and Chest X-ray Images

Thorax disease is a life-threatening disease caused by bacterial infections that occur in the lungs. It could be deadly if not treated at the right time, so early diagnosis of thoracic diseases is vital. The suggested study can assist radiologists in more swiftly diagnosing thorax disorders and in the rapid airport screening of patients with a thorax disease, such as pneumonia. This paper focuses on automatically detecting and localizing thorax disease using chest X-ray images. It provides accurate detection and localization using DenseNet-121 which is foundation of our proposed framework, called Z-Net. The proposed framework utilizes the weighted cross-entropy loss function (W-CEL) that manages class imbalance issue in the ChestX-ray14 dataset, which helped in achieving the highest performance as compared to the previous models. The 112,120 images contained in the ChestX-ray14 dataset (60,412 images are normal, and the rest contain thorax diseases) were preprocessed and then trained for classification and localization. This work uses computer-aided diagnosis (CAD) system that supports development of highly accurate and precise computer-aided systems. We aim to develop a CAD system using a deep learning approach. Our quantitative results show high AUC scores in comparison with the latest research works. The proposed approach achieved the highest mean AUC score of 85.8%. This is the highest accuracy documented in the literature for any related model.

AI-based computer-aided diagnostic system of chest digital tomography synthesis: Demonstrating comparative advantage with X-ray-based AI systems

BACKGROUND

Compared with chest X-ray (CXR) imaging, which is a single image projected from the front of the patient, chest digital tomosynthesis (CDTS) imaging can be more advantageous for lung lesion detection because it acquires multiple images projected from multiple angles of the patient. Various clinical comparative analysis and verification studies have been reported to demonstrate this, but there is no artificial intelligence (AI)-based comparative analysis studies. Existing AI-based computer-aided detection (CAD) systems for lung lesion diagnosis have been developed mainly based on CXR images; however, CAD-based on CDTS, which uses multi-angle images of patients in various directions, has not been proposed and verified for its usefulness compared to CXR-based counterparts.

BACKGROUND AND OBJECTIVE

This study develops and tests a CDTS-based AI CAD system to detect lung lesions to demonstrate performance improvements compared to CXR-based AI CAD.

METHODS

We used multiple (e.g., five) projection images as input for the CDTS-based AI model and a single-projection image as input for the CXR-based AI model to compare and evaluate the performance between models. Multiple/single projection input images were obtained by virtual projection on the three-dimensional (3D) stack of computed tomography (CT) slices of each patient's lungs from which the bed area was removed. These multiple images result from shooting from the front and left and right 30/60∘. The projected image captured from the front was used as the input for the CXR-based AI model. The CDTS-based AI model used all five projected images. The proposed CDTS-based AI model consisted of five AI models that received images in each of the five directions, and obtained the final prediction result through an ensemble of five models. Each model used WideResNet-50. To train and evaluate CXR- and CDTS-based AI models, 500 healthy data, 206 tuberculosis data, and 242 pneumonia data were used, and three three-fold cross-validation was applied.

RESULTS

The proposed CDTS-based AI CAD system yielded sensitivities of 0.782 and 0.785 and accuracies of 0.895 and 0.837 for the (binary classification) performance of detecting tuberculosis and pneumonia, respectively, against normal subjects. These results show higher performance than the sensitivity of 0.728 and 0.698 and accuracies of 0.874 and 0.826 for detecting tuberculosis and pneumonia through the CXR-based AI CAD, which only uses a single projection image in the frontal direction. We found that CDTS-based AI CAD improved the sensitivity of tuberculosis and pneumonia by 5.4% and 8.7% respectively, compared to CXR-based AI CAD without loss of accuracy.

CONCLUSIONS

This study comparatively proves that CDTS-based AI CAD technology can improve performance more than CXR. These results suggest that we can enhance the clinical application of CDTS. Our code is available at https://github.com/kskim-phd/CDTS-CAD-P.

Multi-Label Local to Global Learning: A Novel Learning Paradigm for Chest X-Ray Abnormality Classification

Deep neural network (DNN) approaches have shown remarkable progress in automatic Chest X-rays classification. However, existing methods use a training scheme that simultaneously trains all abnormalities without considering their learning priority. Inspired by the clinical practice of radiologists progressively recognizing more abnormalities and the observation that existing curriculum learning (CL) methods based on image difficulty may not be suitable for disease diagnosis, we propose a novel CL paradigm, named multi-label local to global (ML-LGL). This approach iteratively trains DNN models on gradually increasing abnormalities within the dataset, i,e, from fewer abnormalities (local) to more ones (global). At each iteration, we first build the local category by adding high-priority abnormalities for training, and the abnormality's priority is determined by our three proposed clinical knowledge-leveraged selection functions. Then, images containing abnormalities in the local category are gathered to form a new training set. The model is lastly trained on this set using a dynamic loss. Additionally, we demonstrate the superiority of ML-LGL from the perspective of the model's initial stability during training. Experimental results on three open-source datasets, PLCO, ChestX-ray14 and CheXpert show that our proposed learning paradigm outperforms baselines and achieves comparable results to state-of-the-art methods. The improved performance promises potential applications in multi-label Chest X-ray classification.

Label correlation transformer for automated chest X-ray diagnosis with reliable interpretability

Computer-aided diagnosis of chest X-ray (CXR) images can help reduce the huge workload of radiologists and avoid the inter-observer variability in large-scale early disease screening. Recently, most state-of-the-art studies employ deep learning techniques to address this problem through multi-label classification. However, existing methods still suffer from low classification accuracy and poor interpretability for each diagnostic task. This study aims to propose a novel transformer-based deep learning model for automated CXR diagnosis with high performance and reliable interpretability. We introduce a novel transformer architecture into this problem and utilize the unique query structure of transformer to capture the global and local information of the images and the correlation between labels. In addition, we propose a new loss function to help the model find correlations between the labels in CXR images. To achieve accurate and reliable interpretability, we generate heatmaps using the proposed transformer model and compare with the true pathogenic regions labeled by the physicians. The proposed model achieves a mean AUC of 0.831 on chest X-ray 14 and 0.875 on PadChest dataset, which outperforms existing state-of-the-art methods. The attention heatmaps show that our model could focus on the exact corresponding areas of related truly labeled pathogenic regions. The proposed model effectively improves the performance of CXR multi-label classification and the interpretability of label correlations, thus providing new evidence and methods for automated clinical diagnosis.

Automatic lung disease classification from the chest X-ray images using hybrid deep learning algorithm

The chest X-ray images provide vital information about the congestion cost-effectively. We propose a novel Hybrid Deep Learning Algorithm (HDLA) framework for automatic lung disease classification from chest X-ray images. The model consists of steps including pre-processing of chest X-ray images, automatic feature extraction, and detection. In a pre-processing step, our goal is to improve the quality of raw chest X-ray images using the combination of optimal filtering without data loss. The robust Convolutional Neural Network (CNN) is proposed using the pre-trained model for automatic lung feature extraction. We employed the 2D CNN model for the optimum feature extraction in minimum time and space requirements. The proposed 2D CNN model ensures robust feature learning with highly efficient 1D feature estimation from the input pre-processed image. As the extracted 1D features have suffered from significant scale variations, we optimized them using min-max scaling. We classify the CNN features using the different machine learning classifiers such as AdaBoost, Support Vector Machine (SVM), Random Forest (RM), Backpropagation Neural Network (BNN), and Deep Neural Network (DNN). The experimental results claim that the proposed model improves the overall accuracy by 3.1% and reduces the computational complexity by 16.91% compared to state-of-the-art methods.

Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classification

Multi-label classification (MLC) can attach multiple labels on single image, and has achieved promising results on medical images. But existing MLC methods still face challenging clinical realities in practical use, such as: (1) medical risks arising from misclassification, (2) sample imbalance problem among different diseases, (3) inability to classify the diseases that are not pre-defined (unseen diseases). Here, we design a hybrid label to improve the flexibility of MLC methods and alleviate the sample imbalance problem. Specifically, in the labeled training set, we remain independent labels for high-frequency diseases with enough samples and use a hybrid label to merge low-frequency diseases with fewer samples. The hybrid label can also be used to put unseen diseases in practical use. In this paper, we propose Triplet Attention and Dual-pool Contrastive Learning (TA-DCL) for multi-label medical image classification based on the aforementioned label representation. TA-DCL architecture is a triplet attention network (TAN), which combines category-attention, self-attention and cross-attention together to learn high-quality label embeddings for all disease labels by mining effective information from medical images. DCL includes dual-pool contrastive training (DCT) and dual-pool contrastive inference (DCI). DCT optimizes the clustering centers of label embeddings belonging to different disease labels to improve the discrimination of label embeddings. DCI relieves the error classification of sick cases for reducing the clinical risk and improving the ability to detect unseen diseases by contrast of differences. TA-DCL is validated on two public medical image datasets, ODIR and NIH-ChestXray14, showing superior performance than other state-of-the-art MLC methods. Code is available at https://github.com/ZhangYH0502/TA-DCL.

Bilateral adaptive graph convolutional network on CT based Covid-19 diagnosis with uncertainty-aware consensus-assisted multiple instance learning

Coronavirus disease (COVID-19) has caused a worldwide pandemic, putting millions of people's health and lives in jeopardy. Detecting infected patients early on chest computed tomography (CT) is critical in combating COVID-19. Harnessing uncertainty-aware consensus-assisted multiple instance learning (UC-MIL), we propose to diagnose COVID-19 using a new bilateral adaptive graph-based (BA-GCN) model that can use both 2D and 3D discriminative information in 3D CT volumes with arbitrary number of slices. Given the importance of lung segmentation for this task, we have created the largest manual annotation dataset so far with 7,768 slices from COVID-19 patients, and have used it to train a 2D segmentation model to segment the lungs from individual slices and mask the lungs as the regions of interest for the subsequent analyses. We then used the UC-MIL model to estimate the uncertainty of each prediction and the consensus between multiple predictions on each CT slice to automatically select a fixed number of CT slices with reliable predictions for the subsequent model reasoning. Finally, we adaptively constructed a BA-GCN with vertices from different granularity levels (2D and 3D) to aggregate multi-level features for the final diagnosis with the benefits of the graph convolution network's superiority to tackle cross-granularity relationships. Experimental results on three largest COVID-19 CT datasets demonstrated that our model can produce reliable and accurate COVID-19 predictions using CT volumes with any number of slices, which outperforms existing approaches in terms of learning and generalisation ability. To promote reproducible research, we have made the datasets, including the manual annotations and cleaned CT dataset, as well as the implementation code, available at https://doi.org/10.5281/zenodo.6361963.

A 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis

In 2020, an experiment testing AI solutions for lung X-ray analysis on a multi-hospital network was conducted. The multi-hospital network linked 178 Moscow state healthcare centers, where all chest X-rays from the network were redirected to a research facility, analyzed with AI, and returned to the centers. The experiment was formulated as a public competition with monetary awards for participating industrial and research teams. The task was to perform the binary detection of abnormalities from chest X-rays. For the objective real-life evaluation, no training X-rays were provided to the participants. This paper presents one of the top-performing AI frameworks from this experiment. First, the framework used two EfficientNets, histograms of gradients, Haar feature ensembles, and local binary patterns to recognize whether an input image represents an acceptable lung X-ray sample, meaning the X-ray is not grayscale inverted, is a frontal chest X-ray, and completely captures both lung fields. Second, the framework extracted the region with lung fields and then passed them to a multi-head DenseNet, where the heads recognized the patient's gender, age and the potential presence of abnormalities, and generated the heatmap with the abnormality regions highlighted. During one month of the experiment from 11.23.2020 to 12.25.2020, 17,888 cases have been analyzed by the framework with 11,902 cases having radiological reports with the reference diagnoses that were unequivocally parsed by the experiment organizers. The performance measured in terms of the area under receiving operator curve (AUC) was 0.77. The AUC for individual diseases ranged from 0.55 for herniation to 0.90 for pneumothorax.

EHDFL: Evolutionary hybrid domain feature learning based on windowed fast Fourier convolution pyramid for medical image classification

With the development of modern medical technology, medical image classification has played an important role in medical diagnosis and clinical practice. Medical image classification algorithms based on deep learning emerge in endlessly, and have achieved amazing results. However, most of these methods ignore the feature representation based on frequency domain, and only focus on spatial features. To solve this problem, we propose a hybrid domain feature learning (HDFL) module based on windowed fast Fourier convolution pyramid, which combines the global features with a wide range of receptive fields in frequency domain and the local features with multiple scales in spatial domain. In order to prevent frequency leakage, we construct a Windowed Fast Fourier Convolution (WFFC) structure based on Fast Fourier Convolution (FFC). In order to learn hybrid domain features, we combine ResNet, FPN, and attention mechanism to construct a hybrid domain feature learning module. In addition, a super-parametric optimization algorithm is constructed based on genetic algorithm for our classification model, so as to realize the automation of our super-parametric optimization. We evaluated the newly published medical image classification dataset MedMNIST, and the experimental results show that our method can effectively learning the hybrid domain feature information of frequency domain and spatial domain.

PCAN: Pixel-wise classification and attention network for thoracic disease classification and weakly supervised localization

Automatic chest X-ray (CXR) disease classification has drawn increasing public attention as CXR is widely used in thoracic disease diagnosis. Existing classification networks typically employ a global average pooling layer to produce the final feature for the subsequent classifier. This limits the classification performance owing to the characteristics of lesions in CXR images, including small relative sizes, varied absolute sizes, and different occurrence locations. In this study, we propose a pixel-wise classification and attention network (PCAN) to simultaneously perform disease classification and weakly supervised localization, which provides interpretability for disease classification. The PCAN comprises a backbone network for extracting mid-level features, a pixel-wise classification branch (pc-branch) for generating pixel-wise diagnoses, and a pixel-wise attention branch (pa-branch) for producing pixel-wise weights. The pc-branch is capable of explicitly detecting small lesions, and the pa-branch is capable of adaptively focusing on different regions when classifying different thoracic diseases. Then, the pixel-wise diagnoses are multiplied with the pixel-wise weights to obtain the disease localization map, which provides the sizes and locations of lesions in a manner of weakly supervised learning. The final image-wise diagnosis is obtained by summing up the disease localization map at the spatial dimension. Comprehensive experiments conducted on the ChestX-ray14 and CheXpert datasets demonstrate the effectiveness of the proposed PCAN, which has great potential for thoracic disease diagnosis and treatment. The source codes are available at https://github.com/fzfs/PCAN.

Dual Encoder-Based Dynamic-Channel Graph Convolutional Network With Edge Enhancement for Retinal Vessel Segmentation

Retinal vessel segmentation with deep learning technology is a crucial auxiliary method for clinicians to diagnose fundus diseases. However, the deep learning approaches inevitably lose the edge information, which contains spatial features of vessels while performing down-sampling, leading to the limited segmentation performance of fine blood vessels. Furthermore, the existing methods ignore the dynamic topological correlations among feature maps in the deep learning framework, resulting in the inefficient capture of the channel characterization. To address these limitations, we propose a novel dual encoder-based dynamic-channel graph convolutional network with edge enhancement (DE-DCGCN-EE) for retinal vessel segmentation. Specifically, we first design an edge detection-based dual encoder to preserve the edge of vessels in down-sampling. Secondly, we investigate a dynamic-channel graph convolutional network to map the image channels to the topological space and synthesize the features of each channel on the topological map, which solves the limitation of insufficient channel information utilization. Finally, we study an edge enhancement block, aiming to fuse the edge and spatial features in the dual encoder, which is beneficial to improve the accuracy of fine blood vessel segmentation. Competitive experimental results on five retinal image datasets validate the efficacy of the proposed DE-DCGCN-EE, which achieves more remarkable segmentation results against the other state-of-the-art methods, indicating its potential clinical application.

GSTCNet: Gated spatio-temporal correlation network for stroke mortality prediction

Stroke continues to be the most common cause of death in China. It has great significance for mortality prediction for stroke patients, especially in terms of analyzing the complex interactions between non-negligible factors. In this paper, we present a gated spatio-temporal correlation network (GSTCNet) to predict the one-year post-stroke mortality. Based on the four categories of risk factors: vascular event, chronic disease, medical usage and surgery, we designed a gated correlation graph convolution kernel to capture spatial features and enhance the spatial correlation between feature categories. Bi-LSTM represents the temporal features of five timestamps. The novel gated correlation attention mechanism is then connected to the Bi-LSTM to realize the comprehensive mining of spatio-temporal correlations. Using the data on 2275 patients obtained from the neurology department of a local hospital, we constructed a series of sequential experiments. The experimental results show that the proposed model achieves competitive results on each evaluation metric, reaching an AUC of 89.17%, a precision of 97.75%, a recall of 95.33% and an F1-score of 95.19%. The interpretability analysis of the feature categories and timestamps also verified the potential application value of the model for stroke.

Knowledge Graph Applications in Medical Imaging Analysis: A Scoping Review

Background

There is an increasing trend to represent domain knowledge in structured graphs, which provide efficient knowledge representations for many downstream tasks. Knowledge graphs are widely used to model prior knowledge in the form of nodes and edges to represent semantically connected knowledge entities, which several works have adopted into different medical imaging applications.

Methods

We systematically searched over five databases to find relevant articles that applied knowledge graphs to medical imaging analysis. After screening, evaluating, and reviewing the selected articles, we performed a systematic analysis.

Results

We looked at four applications in medical imaging analysis, including disease classification, disease localization and segmentation, report generation, and image retrieval. We also identified limitations of current work, such as the limited amount of available annotated data and weak generalizability to other tasks. We further identified the potential future directions according to the identified limitations, including employing semisupervised frameworks to alleviate the need for annotated data and exploring task-agnostic models to provide better generalizability.

Conclusions

We hope that our article will provide the readers with aggregated documentation of the state-of-the-art knowledge graph applications for medical imaging to encourage future research.

A metric learning-based method using graph neural network for pancreatic cystic neoplasm classification from CTs

PURPOSE

Pancreatic cystic neoplasms (PCNs) are relatively rare neoplasms and difficult to be classified preoperatively. Ordinary deep learning methods have great potential to provide support for doctors in PCNs classification but require a quantity of labeled samples and exact segmentation of neoplasm. The proposed metric learning-based method using graph neural network aims to overcome the limitations brought by small and imbalanced dataset and get fast and accurate PCNs classification result from computed tomography (CT) images.

METHODS

The proposed framework applies graph neural network (GNN). GNNs perform well in fusing information and modeling relational data and get better results on dataset with small size. Based on metric learning strategy, model learns distance from the data. The similarity-based algorithm enhances the classification performance, and more characteristic information is found. We use a convolutional neural network (CNN) to extract features from given images. Then GNN is used to find the similarity between each two feature vectors and complete the classification. Several subtasks consisting of randomly selected images are established to improve generalization of the model. The experiments are carried out on the dataset provided by Huashan Hospital. The dataset is labeled by postoperative pathological analysis and contains ROI information calibrated by experts. We set two tasks based on the dataset: benign or malignant diagnosis of PCNs and classification of specific types.

RESULTS

Our model shows good performance on the 2 tasks with accuracies of 88.926% and 74.497%. The comparison of different methods' F1 scores in the benign or malignant diagnosis shows the proposed GNN-based method effectively reduces the negative impact brought by imbalanced dataset, which is also verified by the macro-average comparison in the 4-class classification task.

CONCLUSIONS

Compared with existing models, the proposed GNN-based model shows better performance in terms of imbalanced dataset with small size while reducing labeling cost. The result provides a possibility for its application into the computer aided diagnosis of PCNs. This article is protected by copyright. All rights reserved.

Multiparametric Magnetic Resonance Imaging Information Fusion Using Graph Convolutional Network for Glioma Grading

Accurate preoperative glioma grading is essential for clinical decision-making and prognostic evaluation. Multiparametric magnetic resonance imaging (mpMRI) serves as an important diagnostic tool for glioma patients due to its superior performance in describing noninvasively the contextual information in tumor tissues. Previous studies achieved promising glioma grading results with mpMRI data utilizing a convolutional neural network (CNN)-based method. However, these studies have not fully exploited and effectively fused the rich tumor contextual information provided in the magnetic resonance (MR) images acquired with different imaging parameters. In this paper, a novel graph convolutional network (GCN)-based mpMRI information fusion module (named MMIF-GCN) is proposed to comprehensively fuse the tumor grading relevant information in mpMRI. Specifically, a graph is constructed according to the characteristics of mpMRI data. The vertices are defined as the glioma grading features of different slices extracted by the CNN, and the edges reflect the distances between the slices in a 3D volume. The proposed method updates the information in each vertex considering the interaction between adjacent vertices. The final glioma grading is conducted by combining the fused information in all vertices. The proposed MMIF-GCN module can introduce an additional nonlinear representation learning step in the process of mpMRI information fusion while maintaining the positional relationship between adjacent slices. Experiments were conducted on two datasets, that is, a public dataset (named BraTS2020) and a private one (named GliomaHPPH2018). The results indicate that the proposed method can effectively fuse the grading information provided in mpMRI data for better glioma grading performance.

Research on Chest Disease Recognition Based on Deep Hierarchical Learning Algorithm

Chest X-ray has become one of the most common ways in diagnostic radiology exams, and this technology assists expert radiologists with finding the patients at potential risk of cardiopathy and lung diseases. However, it is still a challenge for expert radiologists to assess thousands of cases in a short period so that deep learning methods are introduced to tackle this problem. Since the diseases have correlations with each other and have hierarchical features, the traditional classification scheme could not achieve a good performance. In order to extract the correlation features among the diseases, some GCN-based models are introduced to combine the features extracted from the images to make prediction. This scheme can work well with the high quality of image features, so backbone with high computation cost plays a vital role in this scheme. However, a fast prediction in diagnostic radiology is also needed especially in case of emergency or region with low computation facilities, so we proposed an efficient convolutional neural network with GCN, which is named SGGCN, to meet the need of efficient computation and considerable accuracy. SGGCN used SGNet-101 as backbone, which is built by ShuffleGhost Block (Huang et al., 2021) to extract features with a low computation cost. In order to make sufficient usage of the information in GCN, a new GCN architecture is designed to combine information from different layers together in GCNM module so that we can utilize various hierarchical features and meanwhile make the GCN scheme faster. The experiment on CheXPert datasets illustrated that SGGCN achieves a considerable performance. Compared with GCN and ResNet-101 (He et al., 2015) backbone (test AUC 0.8080, parameters 4.7M and FLOPs 16.0B), the SGGCN achieves 0.7831 (-3.08%) test AUC with parameters 1.2M (-73.73%) and FLOPs 3.1B (-80.82%), where GCN with MobileNet (Sandler and Howard, 2018) backbone achieves 0.7531 (-6.79%) test AUC with parameters 0.5M (-88.46%) and FLOPs 0.66B (-95.88%).

Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future

With the advances of data-driven machine learning research, a wide variety of prediction problems have been tackled. It has become critical to explore how machine learning and specifically deep learning methods can be exploited to analyse healthcare data. A major limitation of existing methods has been the focus on grid-like data; however, the structure of physiological recordings are often irregular and unordered, which makes it difficult to conceptualise them as a matrix. As such, graph neural networks have attracted significant attention by exploiting implicit information that resides in a biological system, with interacting nodes connected by edges whose weights can be determined by either temporal associations or anatomical junctions. In this survey, we thoroughly review the different types of graph architectures and their applications in healthcare. We provide an overview of these methods in a systematic manner, organized by their domain of application including functional connectivity, anatomical structure, and electrical-based analysis. We also outline the limitations of existing techniques and discuss potential directions for future research.

Deep learning for chest X-ray analysis: A survey

Recent advances in deep learning have led to a promising performance in many medical image analysis tasks. As the most commonly performed radiological exam, chest radiographs are a particularly important modality for which a variety of applications have been researched. The release of multiple, large, publicly available chest X-ray datasets in recent years has encouraged research interest and boosted the number of publications. In this paper, we review all studies using deep learning on chest radiographs published before March 2021, categorizing works by task: image-level prediction (classification and regression), segmentation, localization, image generation and domain adaptation. Detailed descriptions of all publicly available datasets are included and commercial systems in the field are described. A comprehensive discussion of the current state of the art is provided, including caveats on the use of public datasets, the requirements of clinically useful systems and gaps in the current literature.

Part-Aware Mask-Guided Attention for Thorax Disease Classification

Thorax disease classification is a challenging task due to complex pathologies and subtle texture changes, etc. It has been extensively studied for years largely because of its wide application in computer-aided diagnosis. Most existing methods directly learn global feature representations from whole Chest X-ray (CXR) images, without considering in depth the richer visual cues lying around informative local regions. Thus, these methods often produce sub-optimal thorax disease classification performance because they ignore the very informative pathological changes around organs. In this paper, we propose a novel Part-Aware Mask-Guided Attention Network (PMGAN) that learns complementary global and local feature representations from all-organ region and multiple single-organ regions simultaneously for thorax disease classification. Specifically, multiple innovative soft attention modules are designed to progressively guide feature learning toward the global informative regions of whole CXR image. A mask-guided attention module is designed to further search for informative regions and visual cues within the all-organ or single-organ images, where attention is elegantly regularized by automatically generated organ masks and without introducing computation during the inference stage. In addition, a multi-task learning strategy is designed, which effectively maximizes the learning of complementary local and global representations. The proposed PMGAN has been evaluated on the ChestX-ray14 dataset and the experimental results demonstrate its superior thorax disease classification performance against the state-of-the-art methods.