A Super-resolution Guided Network for Improving Automated Thyroid Nodule Segmentation

Federated learning via multi-attention guided UNet for thyroid nodule segmentation of ultrasound images

Accurate segmentation of thyroid nodules is essential for early screening and diagnosis, but it can be challenging due to the nodules' varying sizes and positions. To address this issue, we propose a multi-attention guided UNet (MAUNet) for thyroid nodule segmentation. We use a multi-scale cross attention (MSCA) module for initial image feature extraction. By integrating interactions between features at different scales, the impact of thyroid nodule shape and size on the segmentation results has been reduced. Additionally, we incorporate a dual attention (DA) module into the skip-connection step of the UNet network, which promotes information exchange and fusion between the encoder and decoder. To test the model's robustness and effectiveness, we conduct the extensive experiments on multi-center ultrasound images provided by 17 local hospitals. The model is trained using the federal learning mechanism to ensure privacy protection. The experimental results show that the Dice scores of the model on the data sets from the three centers are 0.908, 0.912 and 0.887, respectively. Compared to existing methods, our method demonstrates higher generalization ability on multi-center datasets and achieves better segmentation results.

The study on ultrasound image classification using a dual-branch model based on Resnet50 guided by U-net segmentation results

In recent years, the incidence of nodular thyroid diseases has been increasing annually. Ultrasonography has become a routine diagnostic tool for thyroid nodules due to its high real-time capabilities and low invasiveness. However, thyroid images obtained from current ultrasound tests often have low resolution and are plagued by significant noise interference. Regional differences in medical conditions and varying levels of physician experience can impact the accuracy and efficiency of diagnostic results. With the advancement of deep learning technology, deep learning models are used to identify whether a nodule in a thyroid ultrasound image is benign or malignant. This helps to close the gap between doctors' experience and equipment differences, improving the accuracy of the initial diagnosis of thyroid nodules. To cope with the problem that thyroid ultrasound images contain complex background and noise as well as poorly defined local features. in this paper, we first construct an improved ResNet50 classification model that uses a two-branch input and incorporates a global attention lightening module. This model is used to improve the accuracy of benign and malignant nodule classification in thyroid ultrasound images and to reduce the computational effort due to the two-branch structure.We constructed a U-net segmentation model incorporating our proposed ACR module, which uses hollow convolution with different dilation rates to capture multi-scale contextual information for feature extraction of nodules in thyroid ultrasound images and uses the results of the segmentation task as an auxiliary branch for the classification task to guide the classification model to focus on the lesion region more efficiently in the case of weak local features. The classification model is guided to focus on the lesion region more efficiently, and the classification and segmentation sub-networks are respectively improved specifically for this study, which is used to improve the accuracy of classifying the benign and malignant nature of the nodules in thyroid ultrasound images. The experimental results show that the four evaluation metrics of accuracy, precision, recall, and f1 of the improved model are 96.01%, 93.3%, 98.8%, and 96.0%, respectively. The improvements were 5.7%, 1.6%, 13.1%, and 7.4%, respectively, compared with the baseline classification model.

Deblurring masked image modeling for ultrasound image analysis

Recently, large pretrained vision foundation models based on masked image modeling (MIM) have attracted unprecedented attention and achieved remarkable performance across various tasks. However, the study of MIM for ultrasound imaging remains relatively unexplored, and most importantly, current MIM approaches fail to account for the gap between natural images and ultrasound, as well as the intrinsic imaging characteristics of the ultrasound modality, such as the high noise-to-signal ratio. In this paper, motivated by the unique high noise-to-signal ratio property in ultrasound, we propose a deblurring MIM approach specialized to ultrasound, which incorporates a deblurring task into the pretraining proxy task. The incorporation of deblurring facilitates the pretraining to better recover the subtle details within ultrasound images that are vital for subsequent downstream analysis. Furthermore, we employ a multi-scale hierarchical encoder to extract both local and global contextual cues for improved performance, especially on pixel-wise tasks such as segmentation. We conduct extensive experiments involving 280,000 ultrasound images for the pretraining and evaluate the downstream transfer performance of the pretrained model on various disease diagnoses (nodule, Hashimoto's thyroiditis) and task types (classification, segmentation). The experimental results demonstrate the efficacy of the proposed deblurring MIM, achieving state-of-the-art performance across a wide range of downstream tasks and datasets. Overall, our work highlights the potential of deblurring MIM for ultrasound image analysis, presenting an ultrasound-specific vision foundation model.

Via Multi-attention Guided UNet for Thyroid Nodule Segmentation of Ultrasound Images

Accurate segmentation of thyroid nodules is essential for early screening and diagnosis, but it can be challenging due to the nodules' varying sizes and positions. To address this issue, we propose a multi-attention guided UNet (MAUNet) for thyroid nodule segmentation. We use a multi-scale cross attention (MSCA) module for initial image feature extraction. By integrating interactions between features at different scales, the impact of thyroid nodule shape and size on the segmentation results has been reduced. Additionally, we incorporate a dual attention (DA) module into the skip-connection step of the UNet network, which promotes information exchange and fusion between the encoder and decoder. To test the model's robustness and effectiveness, we conduct the extensive experiments on multi-center ultrasound images provided by 17 hospitals. The experimental results show that our proposed method outperforms existing deep learning methods, which provides a new research direction for detecting thyroid nodules.

Artificial intelligence assisted diagnosis of early tc markers and its application

Thyroid cancer (TC) is a common endocrine malignancy with an increasing incidence worldwide. Early diagnosis is particularly important for TC patients, because it allows patients to receive treatment as early as possible. Artificial intelligence (AI) provides great advantages for complex healthcare systems by analyzing big data based on machine learning. Nowadays, AI is widely used in the early diagnosis of cancer such as TC. Ultrasound detection and fine needle aspiration biopsy are the main methods for early diagnosis of TC. AI has been widely used in the detection of malignancy in thyroid nodules by ultrasound images, cytopathology images and molecular markers. It shows great potential in auxiliary medical diagnosis. The latest clinical trial has shown that the performance of AI models matches with the diagnostic efficiency of experienced clinicians, and more efficient AI tools will be developed in the future. Therefore, in this review, we summarized the recent advances in the application of AI algorithms in assessing the risk of malignancy in thyroid nodules. The objective of this review was to provide a data base for the clinical use of AI-assisted diagnosis in TC, as well as to provide new ideas for the next generation of AI-assisted diagnosis in TC.

GLFNet: Global-local fusion network for the segmentation in ultrasound images

Ultrasound imaging, as a portable and radiation-free modality, presents challenges for accurate segmentation due to the variability of lesions and the similar intensity values of surrounding tissues. Current deep learning approaches leverage convolution for extracting local features and self-attention for handling global dependencies. However, traditional CNNs are spatially local, and Vision Transformers lack image specific bias and are computationally demanding. In response, we propose the Global-Local Fusion Network (GLFNet), a hybrid structure addressing the limitations of both CNNs and Vision Transformers. The GLFNet, featuring Global-Local Fusion Blocks (GLFBlocks), integrates global semantic information with local details to improve segmentation. Each GLFBlock comprises Global and Local Branches for feature extraction in parallel. Within the Global and Local Branches, we introduce the Self-Attention Convolution Fusion Block (SACFBlock), which includes a Spatial-Attention Module and Channel-Attention Module. Experimental results show that our proposed GLFNet surpasses its counterparts in the segmentation tasks, achieving the overall best results with an mIoU of 79.58% and Dice coefficient of 74.62% in the DDTI dataset, an mIoU of 76.61% and Dice coefficient of 71.04% in the BUSI dataset, and an mIoU of 86.77% and Dice coefficient of 87.38% in the BUID dataset. The fusion of local and global features contributes to enhanced performance, making GLFNet a promising approach for ultrasound image segmentation.

Segmentation of Substantia Nigra in Brain Parenchyma Sonographic Images Using Deep Learning

Currently, Parkinson's Disease (PD) is diagnosed primarily based on symptoms by experts clinicians. Neuroimaging exams represent an important tool to confirm the clinical diagnosis. Among them, Brain Parenchyma Sonography (BPS) is used to evaluate the hyperechogenicity of Substantia Nigra (SN), found in more than 90% of PD patients. In this article, we exploit a new dataset of BPS images to investigate an automatic segmentation approach for SN that can increase the accuracy of the exam and its practicability in clinical routine. This study achieves state-of-the-art performance in SN segmentation of BPS images. Indeed, it is found that the modified U-Net network scores a Dice coefficient of 0.859 ± 0.037. The results presented in this study demonstrate the feasibility and usefulness of SN automatic segmentation in BPS medical images, to the point that this study can be considered as the first stage of the development of an end-to-end CAD (Computer Aided Detection) system. Furthermore, the used dataset, which will be further enriched in the future, has proven to be very effective in supporting the training of CNNs and may pave the way for future studies in the field of CAD applied to PD.

STMMOT: Advancing multi-object tracking through spatiotemporal memory networks and multi-scale attention pyramids

Multi-object Tracking (MOT) is very important in human surveillance, sports analytics, autonomous driving, and cooperative robots. Current MOT methods do not perform well in non-uniform movements, occlusion and appearance-reappearance scenarios. We introduce a comprehensive MOT method that seamlessly merges object detection and identity linkage within an end-to-end trainable framework, designed with the capability to maintain object links over a long period of time. Our proposed model, named STMMOT, is architectured around 4 key modules: (1) Candidate proposal creation network, generates object proposals via vision-Transformer encoder-decoder architecture; (2) Scale variant pyramid, progressive pyramid structure to learn the self-scale and cross-scale similarities in multi-scale feature maps; (3) Spatio-temporal memory encoder, extracting the essential information from the memory associated with each object under tracking; and (4) Spatio-temporal memory decoder, simultaneously resolving the tasks of object detection and identity association for MOT. Our system leverages a robust spatio-temporal memory module that retains extensive historical object state observations and effectively encodes them using an attention-based aggregator. The uniqueness of STMMOT resides in representing objects as dynamic query embeddings that are updated continuously, which enables the prediction of object states with an attention mechanism and eradicates the need for post-processing. Experimental results show that STMMOT archives scores of 79.8 and 78.4 for IDF1, 79.3 and 74.1 for MOTA, 73.2 and 69.0 for HOTA, 61.2 and 61.5 for AssA, and maintained an ID switch count of 1529 and 1264 on MOT17 and MOT20, respectively. When evaluated on MOT20, it scored 78.4 in IDF1, 74.1 in MOTA, 69.0 in HOTA, and 61.5 in AssA, and kept the ID switch count to 1264. Compared with the previous best TransMOT, STMMOT achieves around a 4.58% and 4.25% increase in IDF1, and ID switching reduction to 5.79% and 21.05% on MOT17 and MOT20, respectively.

Deep multi-task learning for nephropathy diagnosis on immunofluorescence images

BACKGROUND AND OBJECTIVE

As an advanced technique, immunofluorescence (IF) is one of the most widely-used medical image for nephropathy diagnosis, due to its ease of acquisition with low cost. In practice, the clinically collected IF images are commonly corrupted by blurs at different degrees, mainly because of the inaccurate focus at the acquisition stage. Although deep neural network (DNN) methods achieve the great success in nephropathy diagnosis, their performance dramatically drops over the blurred IF images. This significantly limits the potential of leveraging the advanced DNN techniques in real-world nephropathy diagnosis scenarios.

METHODS

This paper first establishes two IF databases with synthetic blurs (IFVB) and real-world blurs (Real-IF) for nephropathy diagnosis, respectively, including 1,659 patients and 6,521 IF images with various degrees of blurs. According to the analysis on these two databases, we propose a deep hierarchical multi-task learning based nephropathy diagnosis (DeepMT-ND) method to bridge the gap between the low-level vision and high-level medical tasks. Specifically, DeepMT-ND simultaneously handles the main task of automatic nephropathy diagnosis, as well as the auxiliary tasks of image quality assessment (IQA) and de-blurring.

RESULTS

Extensive experiments show the superiority of our DeepMT-ND in terms of diagnosis accuracy and generalization ability. For instance, our method performs better than nephrologists with at least 15.4% and 6.5% accuracy improvements in IFVB and Real-IF, respectively. Meanwhile, our method also achieves comparable performance in two auxiliary tasks of IQA and de-blurring on blurred IF images.

CONCLUSIONS

In this paper, we propose a new DeepMT-ND method for nephropathy diagnosis on blurred IF images. The proposed hierarchical multi-task learning framework provides the new scope to narrow the gap between the low-level vision and high-level medical tasks, and will contribute to nephropathy diagnosis in clinical scenarios. The diagnosis accuracy and generalization ability of DeepMT-ND are experimentally verified to be effective over both synthetic and real-world databases.

[An Improved Object Detection Algorithm for Thyroid Nodule Ultrasound Image Based on Faster R-CNN]

Objective

To propose an improved algorithm for thyroid nodule object detection based on Faster R-CNN so as to improve the detection precision of thyroid nodules in ultrasound images.

Methods

The algorithm used ResNeSt50 combined with deformable convolution (DC) as the backbone network to improve the detection effect of irregularly shaped nodules. Feature pyramid networks (FPN) and Region of Interest (RoI) Align were introduced in the back of the trunk network. The former was used to reduce missed or mistaken detection of thyroid nodules, and the latter was used to improve the detection precision of small nodules. To improve the generalization ability of the model, parameters were updated during backpropagation with an optimizer improved by Sharpness-Aware Minimization (SAM).

Results

In this experiment, 6 261 thyroid ultrasound images from the Affiliated Hospital of Xuzhou Medical University and the First Hospital of Nanjing were used to compare and evaluate the effectiveness of the improved algorithm. According to the findings, the algorithm showed optimization effect to a certain degree, with the AP50 of the final test set being as high as 97.4% and AP@50:5:95 also showing a 10.0% improvement compared with the original model. Compared with both the original model and the existing models, the improved algorithm had higher detection precision and improved capacity to detect thyroid nodules with better accuracy and precision. In particular, the improved algorithm had a higher recall rate under the requirement of lower detection frame precision.

Conclusion

The improved method proposed in the study is an effective object detection algorithm for thyroid nodules and can be used to detect thyroid nodules with accuracy and precision.

Segmentation of thyroid glands and nodules in ultrasound images using the improved U-Net architecture

BACKGROUND

Identifying thyroid nodules' boundaries is crucial for making an accurate clinical assessment. However, manual segmentation is time-consuming. This paper utilized U-Net and its improved methods to automatically segment thyroid nodules and glands.

METHODS

The 5822 ultrasound images used in the experiment came from two centers, 4658 images were used as the training dataset, and 1164 images were used as the independent mixed test dataset finally. Based on U-Net, deformable-pyramid split-attention residual U-Net (DSRU-Net) by introducing ResNeSt block, atrous spatial pyramid pooling, and deformable convolution v3 was proposed. This method combined context information and extracts features of interest better, and had advantages in segmenting nodules and glands of different shapes and sizes.

RESULTS

DSRU-Net obtained 85.8% mean Intersection over Union, 92.5% mean dice coefficient and 94.1% nodule dice coefficient, which were increased by 1.8%, 1.3% and 1.9% compared with U-Net.

CONCLUSIONS

Our method is more capable of identifying and segmenting glands and nodules than the original method, as shown by the results of correlational studies.

Saliency-CCE: Exploiting colour contextual extractor and saliency-based biomedical image segmentation

Biomedical image segmentation is one critical component in computer-aided system diagnosis. However, various non-automatic segmentation methods are usually designed to segment target objects with single-task driven, ignoring the potential contribution of multi-task, such as the salient object detection (SOD) task and the image segmentation task. In this paper, we propose a novel dual-task framework for white blood cell (WBC) and skin lesion (SL) saliency detection and segmentation in biomedical images, called Saliency-CCE. Saliency-CCE consists of a preprocessing of hair removal for skin lesions images, a novel colour contextual extractor (CCE) module for the SOD task and an improved adaptive threshold (AT) paradigm for the image segmentation task. In the SOD task, we perform the CCE module to extract hand-crafted features through a novel colour channel volume (CCV) block and a novel colour activation mapping (CAM) block. We first exploit the CCV block to generate a target object's region of interest (ROI). After that, we employ the CAM block to yield a refined salient map as the final salient map from the extracted ROI. We propose a novel adaptive threshold (AT) strategy in the segmentation task to automatically segment the WBC and SL from the final salient map. We evaluate our proposed Saliency-CCE on the ISIC-2016, the ISIC-2017, and the SCISC datasets, which outperform representative state-of-the-art SOD and biomedical image segmentation approaches. Our code is available at https://github.com/zxg3017/Saliency-CCE.