Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks

Effective data selection via deep learning processes and corresponding learning strategies in ultrasound image classification

In this study, we propose a novel approach to enhancing transfer learning by optimizing data selection through deep learning techniques and corresponding innovative learning strategies. This method is particularly beneficial when the available dataset has reached its limit and cannot be further expanded. Our approach focuses on maximizing the use of existing data to improve learning outcomes which offers an effective solution for data-limited applications in medical imaging classification. The proposed method consists of two stages. In the first stage, an original network performs the initial classification. When the original network exhibits low confidence in its predictions, ambiguous classifications are passed to a secondary decision-making step involving a newly trained network, referred to as the True network. The True network shares the same architecture as the original network but is trained on a subset of the original dataset that is selected based on consensus among multiple independent networks. It is then used to verify the classification results of the original network, identifying and correcting any misclassified images. To evaluate the effectiveness of our approach, we conducted experiments using thyroid nodule ultrasound images with the ResNet101 and Vision Transformer architectures along with eleven other pre-trained neural networks. The proposed method led to performance improvements across all five key metrics, accuracy, sensitivity, specificity, F1-score, and AUC, compared to using only the original or True networks in ResNet101. Additionally, the True network showed strong performance when applied to the Vision Transformer and similar enhancements were observed across multiple convolutional neural network architectures. Furthermore, to assess the robustness and adaptability of our method across different medical imaging modalities, we applied it to dermoscopic images and observed similar performance enhancements. These results provide evidence of the effectiveness of our approach in improving transfer learning-based medical image classification without requiring additional training data.

Quantitative analysis of studies that use artificial intelligence on thyroid cancer: a 20-year bibliometric analysis

In recent years, with the rapid advancement of computer science, artificial intelligence has found extensive applications and has been the subject of significant research within the healthcare industry, particularly in areas such as medical imaging, diagnostics, biomedical engineering, and health data analytics. Artificial intelligence has also made considerable inroads in the diagnosis and treatment of thyroid cancer. This study aims to evaluate the progress, current hotspots, and potential future directions of research on artificial intelligence in the field of thyroid cancer through a bibliometric analysis. This study retrieved literature on the application of artificial intelligence in thyroid cancer from 2004 to 2024 from the Web of Science Core Collection (WoSCC) database. A retrospective bibliometric analysis and visualization study of the filtered data were conducted using VOSviewer, CiteSpace, and the Bibliometrix package in R software. A total of 956 articles from 70 countries/regions were included. China had the highest number of publications, with Shanghai Jiao Tong University (China) being the most prolific research institution. The most prolific author was Wei, X. (n=14), while Haugen, B. R. was the most co-cited author (n=297). The Frontiers in Oncology (35 articles, IF=3.5, Q1) was the most frequently publishing journal, and Thyroid (cited 1,705 times) was the most co-cited journal. Keywords such as 'ultrasound,' 'deep learning,' and 'diagnosis' indicate research hotspots in this field. This study provides a comprehensive exposition of the current advancements, emerging trends, and future directions of artificial intelligence in thyroid cancer research. It serves as a valuable resource for clinicians and researchers, offering a systematic understanding of key focal areas in the field, thereby assisting in the identification and determination of future research trajectories.

A Dual-Branch Cross-Modality-Attention Network for Thyroid Nodule Diagnosis Based on Ultrasound Images and Contrast-Enhanced Ultrasound Videos

Contrast-enhanced ultrasound (CEUS) has been extensively employed as an imaging modality in thyroid nodule diagnosis due to its capacity to visualise the distribution and circulation of micro-vessels in organs and lesions in a non-invasive manner. However, current CEUS-based thyroid nodule diagnosis methods suffered from: 1) the blurred spatial boundaries between nodules and other anatomies in CEUS videos, and 2) the insufficient representations of the local structural information of nodule tissues by the features extracted only from CEUS videos. In this paper, we propose a novel dual-branch network with a cross-modality-attention mechanism for thyroid nodule diagnosis by integrating the information from tow related modalities, i.e., CEUS videos and ultrasound image. The mechanism has two parts: US-attention-from-CEUS transformer (UAC-T) and CEUS-attention-from-US transformer (CAU-T). As such, this network imitates the manner of human radiologists by decomposing the diagnosis into two correlated tasks: 1) the spatio-temporal features extracted from CEUS are hierarchically embedded into the spatial features extracted from US with UAC-T for the nodule segmentation; 2) the US spatial features are used to guide the extraction of the CEUS spatio-temporal features with CAU-T for the nodule classification. The two tasks are intertwined in the dual-branch end-to-end network and optimized with the multi-task learning (MTL) strategy. The proposed method is evaluated on our collected thyroid US-CEUS dataset. Experimental results show that our method achieves the classification accuracy of 86.92%, specificity of 66.41%, and sensitivity of 97.01%, outperforming the state-of-the-art methods. As a general contribution in the field of multi-modality diagnosis of diseases, the proposed method has provided an effective way to combine static information with its related dynamic information, improving the quality of deep learning based diagnosis with an additional benefit of explainability.

Improved patient identification by incorporating symptom severity in deep learning using neuroanatomic images in first episode schizophrenia

Brain alterations associated with illness severity in schizophrenia remain poorly understood. Establishing linkages between imaging biomarkers and symptom expression may enhance mechanistic understanding of acute psychotic illness. Constructing models using MRI and clinical features together to maximize model validity may be particularly useful for these purposes. A multi-task deep learning model for standard case/control recognition incorporated with psychosis symptom severity regression was constructed with anatomic MRI collected from 286 patients with drug-naïve first-episode schizophrenia and 330 healthy controls from two datasets, and validated with an independent dataset including 40 first-episode schizophrenia. To evaluate the contribution of regression to the case/control recognition, a single-task classification model was constructed. Performance of unprocessed anatomical images and of predefined imaging features obtained using voxel-based morphometry (VBM) and surface-based morphometry (SBM), were examined and compared. Brain regions contributing to the symptom severity regression and illness identification were identified. Models developed with unprocessed images achieved greater group separation than either VBM or SBM measurements, differentiating schizophrenia patients from healthy controls with a balanced accuracy of 83.0% with sensitivity = 76.1% and specificity = 89.0%. The multi-task model also showed superior performance to single-task classification model without considering clinical symptoms. These findings showed high replication in the site-split validation and external validation analyses. Measurements in parietal, occipital and medial frontal cortex and bilateral cerebellum had the greatest contribution to the multi-task model. Incorporating illness severity regression in pattern recognition algorithms, our study developed an MRI-based model that was of high diagnostic value in acutely ill schizophrenia patients, highlighting clinical relevance of the model.

Deep learning methods for improving the accuracy and efficiency of pathological image analysis

This study presents a novel integration of two advanced deep learning models, U-Net and EfficientNetV2, to achieve high-precision segmentation and rapid classification of pathological images. A key innovation is the development of a new heatmap generation algorithm, which leverages meticulous image preprocessing, data enhancement strategies, ensemble learning, attention mechanisms, and deep feature fusion techniques. This algorithm not only produces highly accurate and interpretatively rich heatmaps but also significantly improves the accuracy and efficiency of pathological image analysis. Unlike existing methods, our approach integrates these advanced techniques into a cohesive framework, enhancing its ability to reveal critical features in pathological images. Rigorous experimental validation demonstrated that our algorithm excels in key performance indicators such as accuracy, recall rate, and processing speed, underscoring its potential for broader applications in pathological image analysis and beyond.

High-level feature-guided attention optimized neural network for neonatal lateral ventricular dilatation prediction

BACKGROUND

Periventricular-intraventricular hemorrhage can lead to posthemorrhagic ventricular dilatation or even posthemorrhagic hydrocephalus if not detected promptly. Sequential cranial ultrasound scans are typically used for their diagnoses. Nonetheless, manual image audit has numerous disadvantages.

PURPOSE

This study aimed to develop a predictive model utilizing modified inception (MI) and high-level feature-guided attention (HFA) modules for predicting neonatal lateral ventricular dilation via ultrasound images.

METHODS

The MI modules reduced input data sizes and dimensions, while the HFA modules effectively delved into semantic information through supervision from high-level feature images to low-level feature images. The process facilitated the accurate identification of dilated lateral ventricles. A total of 710 neonates, corresponding to 1420 lateral ventricles, were recruited in this study. Each lateral ventricle was captured in two images, one on the parasagittal plane and the other on the coronal plane. The combination of anterior horn width, ventricular index, thalamo-occipital distance, and ventricular height served as the gold standard. A lateral ventricle would be considered dilatated if any of these four indices exceeded its upper reference value. These lateral ventricles were randomly split into training and testing sets at a 7:3 ratio. We evaluated the validity of our proposed approach and its competitors across the coronal plane, parasagittal plane, and overall performance. We also determined the impact of subjects' baseline characteristics on the overall performance of the proposed approach. Additionally, ablation analyses were conducted to ensure the efficacy of the proposed approach.

RESULTS

Our proposed approach achieved the largest Youden index (0.65, 95% CI: 0.58-0.72), DOR (27.11, 95% CI: 15.89-46.26), area under curves (AUC) of receiver operating characteristic curve (ROC) (0.84, 95% CI: 0.80-0.88), and AUC of precision-recall curve (PRC) (0.81, 95% CI: 0.74-0.86) in the overall performance assessment and ablation analyses. Moreover, it boasted the biggest Cramer's V values on the coronal (Cramer's V = 0.488, p < 0.001) and parasagittal (Cramer's V = 0.713, p < 0.001) planes individually. Factors such as left side, male sex, singleton birth, and vaginal delivery were positively correlated with higher performance regarding the proposed algorithm, except for the gestational age.

CONCLUSION

This work provides a novel attention optimized algorithm for rapid and accurate ventricular dilatation predictions. It surpasses the traditional algorithms in terms of validity whether concerning the coronal plane, parasagittal plane, or overall performance. The overall performance of algorithms will be influenced by the baseline characteristics of populations.

Identification method of thyroid nodule ultrasonography based on self-supervised learning dual-branch attention learning framework

Thyroid ultrasound is a widely used diagnostic technique for thyroid nodules in clinical practice. However, due to the characteristics of ultrasonic imaging, such as low image contrast, high noise levels, and heterogeneous features, detecting and identifying nodules remains challenging. In addition, high-quality labeled medical imaging datasets are rare, and thyroid ultrasound images are no exception, posing a significant challenge for machine learning applications in medical image analysis. In this study, we propose a Dual-branch Attention Learning (DBAL) convolutional neural network framework to enhance thyroid nodule detection by capturing contextual information. Leveraging jigsaw puzzles as a pretext task during network training, we improve the network's generalization ability with limited data. Our framework effectively captures intrinsic features in a global-to-local manner. Experimental results involve self-supervised pre-training on unlabeled ultrasound images and fine-tuning using 1216 clinical ultrasound images from a collaborating hospital. DBAL achieves accurate discrimination of thyroid nodules, with a 88.5% correct diagnosis rate for malignant and benign nodules and a 93.7% area under the ROC curve. This novel approach demonstrates promising potential in clinical applications for its accuracy and efficiency.

Detecting thyroid nodules along with surrounding tissues and tracking nodules using motion prior in ultrasound videos

Ultrasound examination plays a crucial role in the clinical diagnosis of thyroid nodules. Although deep learning technology has been applied to thyroid nodule examinations, the existing methods all overlook the prior knowledge of nodules moving along a straight line in the video. We propose a new detection model, DiffusionVID-Line, and design a novel tracking algorithm, ByteTrack-Line, both of which fully leverage the prior knowledge of linear motion of nodules in thyroid ultrasound videos. Among them, ByteTrack-Line groups detected nodules, further reducing the workload of doctors and significantly improving their diagnostic speed and accuracy. In DiffusionVID-Line, we propose two new modules: Freq-FPN and Attn-Line. Freq-FPN module is used to extract frequency features, taking advantage of these features to reduce the impact of image blur in ultrasound videos. Based on the standard practice of segmented scanning by doctors, Attn-Line module enhances the attention on targets moving along a straight line, thus improving the accuracy of detection. In ByteTrack-Line, considering the characteristic of linear motion of nodules, we propose the Match-Line association module, which reduces the number of nodule ID switches. In the testing of the detection and tracking datasets, DiffusionVID-Line achieved a mean Average Precision (mAP50) of 74.2 for multiple tissues and 85.6 for nodules, while ByteTrack-Line achieved a Multiple Object Tracking Accuracy (MOTA) of 83.4. Both nodule detection and tracking have achieved state-of-the-art performance.

Deblurring masked image modeling for ultrasound image analysis

Recently, large pretrained vision foundation models based on masked image modeling (MIM) have attracted unprecedented attention and achieved remarkable performance across various tasks. However, the study of MIM for ultrasound imaging remains relatively unexplored, and most importantly, current MIM approaches fail to account for the gap between natural images and ultrasound, as well as the intrinsic imaging characteristics of the ultrasound modality, such as the high noise-to-signal ratio. In this paper, motivated by the unique high noise-to-signal ratio property in ultrasound, we propose a deblurring MIM approach specialized to ultrasound, which incorporates a deblurring task into the pretraining proxy task. The incorporation of deblurring facilitates the pretraining to better recover the subtle details within ultrasound images that are vital for subsequent downstream analysis. Furthermore, we employ a multi-scale hierarchical encoder to extract both local and global contextual cues for improved performance, especially on pixel-wise tasks such as segmentation. We conduct extensive experiments involving 280,000 ultrasound images for the pretraining and evaluate the downstream transfer performance of the pretrained model on various disease diagnoses (nodule, Hashimoto's thyroiditis) and task types (classification, segmentation). The experimental results demonstrate the efficacy of the proposed deblurring MIM, achieving state-of-the-art performance across a wide range of downstream tasks and datasets. Overall, our work highlights the potential of deblurring MIM for ultrasound image analysis, presenting an ultrasound-specific vision foundation model.

A Survey on Explainable Artificial Intelligence (XAI) Techniques for Visualizing Deep Learning Models in Medical Imaging

The combination of medical imaging and deep learning has significantly improved diagnostic and prognostic capabilities in the healthcare domain. Nevertheless, the inherent complexity of deep learning models poses challenges in understanding their decision-making processes. Interpretability and visualization techniques have emerged as crucial tools to unravel the black-box nature of these models, providing insights into their inner workings and enhancing trust in their predictions. This survey paper comprehensively examines various interpretation and visualization techniques applied to deep learning models in medical imaging. The paper reviews methodologies, discusses their applications, and evaluates their effectiveness in enhancing the interpretability, reliability, and clinical relevance of deep learning models in medical image analysis.

Can incorporating image resolution into neural networks improve kidney tumor classification performance in ultrasound images?

Deep learning has been widely used in ultrasound image analysis, and it also benefits kidney ultrasound interpretation and diagnosis. However, the importance of ultrasound image resolution often goes overlooked within deep learning methodologies. In this study, we integrate the ultrasound image resolution into a convolutional neural network and explore the effect of the resolution on diagnosis of kidney tumors. In the process of integrating the image resolution information, we propose two different approaches to narrow the semantic gap between the features extracted by the neural network and the resolution features. In the first approach, the resolution is directly concatenated with the features extracted by the neural network. In the second approach, the features extracted by the neural network are first dimensionally reduced and then combined with the resolution features to form new composite features. We compare these two approaches incorporating the resolution with the method without incorporating the resolution on a kidney tumor dataset of 926 images consisting of 211 images of benign kidney tumors and 715 images of malignant kidney tumors. The area under the receiver operating characteristic curve (AUC) of the method without incorporating the resolution is 0.8665, and the AUCs of the two approaches incorporating the resolution are 0.8926 (P < 0.0001) and 0.9135 (P < 0.0001) respectively. This study has established end-to-end kidney tumor classification systems and has demonstrated the benefits of integrating image resolution, showing that incorporating image resolution into neural networks can more accurately distinguish between malignant and benign kidney tumors in ultrasound images.

Via Multi-attention Guided UNet for Thyroid Nodule Segmentation of Ultrasound Images

Accurate segmentation of thyroid nodules is essential for early screening and diagnosis, but it can be challenging due to the nodules' varying sizes and positions. To address this issue, we propose a multi-attention guided UNet (MAUNet) for thyroid nodule segmentation. We use a multi-scale cross attention (MSCA) module for initial image feature extraction. By integrating interactions between features at different scales, the impact of thyroid nodule shape and size on the segmentation results has been reduced. Additionally, we incorporate a dual attention (DA) module into the skip-connection step of the UNet network, which promotes information exchange and fusion between the encoder and decoder. To test the model's robustness and effectiveness, we conduct the extensive experiments on multi-center ultrasound images provided by 17 hospitals. The experimental results show that our proposed method outperforms existing deep learning methods, which provides a new research direction for detecting thyroid nodules.

Analysis of artificial intelligence in thyroid diagnostics and surgery: A scoping review

BACKGROUND

Artificial Intelligence provides numerous applications in the healthcare sector. The main aim of this study is to evaluate the extent of the current application of artificial intelligence in thyroid diagnostics.

METHODS

Our protocol was based on the Scoping Reviews extension of the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA-ScR). Information was gathered from PubMed, Cochrane, and EMBASE databases and Google Scholar. Eligible studies were published between 2017 and 2022.

RESULTS

The search identified 133 records, after which 18 articles were included in the scoping review. All the publications were journal articles and discussed various ways that specialists in thyroid diagnostics and surgery have utilized artificial intelligence in their practice.

CONCLUSIONS

The development and incorporation of Artificial Intelligence applications in thyroid diagnostics and surgery has been moderate yet promising. However, applications are currently inconsistent and further research is needed to delineate the true benefit and limitations in this field.

A multi-task model for reliable classification of thyroid nodules in ultrasound images

Thyroid nodules are common, and patients with potential malignant lesions are usually diagnosed using ultrasound imaging to determine further treatment options. This study aims to propose a computer-aided diagnosis method for benign and malignant classification of thyroid nodules in ultrasound images. We propose a novel multi-task framework that combines the advantages of dense connectivity, Squeeze-and-Excitation (SE) connectivity, and Atrous Spatial Pyramid Pooling (ASPP) layer to enhance feature extraction. The Dense connectivity is used to optimize feature reuse, the SE connectivity to optimize feature weights, the ASPP layer to fuse feature information, and a multi-task learning framework to adjust the attention of the network. We evaluate our model using a 10-fold cross-validation approach based on our established Thyroid dataset. We assess the performance of our method using six average metrics: accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and AUC, which are 93.49, 95.54, 91.52, 91.63, 95.47, and 96.84%, respectively. Our proposed method outperforms other classification networks in all metrics, achieving optimal performance. We propose a multi-task model, DSMA-Net, for distinguishing thyroid nodules in ultrasound images. This method can further enhance the diagnostic ability of doctors for suspected cancer patients and holds promise for clinical applications.

Automated classification of liver fibrosis stages using ultrasound imaging

BACKGROUND

Ultrasound imaging is the most frequently performed for the patients with chronic hepatitis or liver cirrhosis. However, ultrasound imaging is highly operator dependent and interpretation of ultrasound images is subjective, thus well-trained radiologist is required for evaluation. Automated classification of liver fibrosis could alleviate the shortage of skilled radiologist especially in low-to-middle income countries. The purposed of this study is to evaluate deep convolutional neural networks (DCNNs) for classifying the degree of liver fibrosis according to the METAVIR score using US images.

METHODS

We used ultrasound (US) images from two tertiary university hospitals. A total of 7920 US images from 933 patients were used for training/validation of DCNNs. All patient were underwent liver biopsy or hepatectomy, and liver fibrosis was categorized based on pathology results using the METAVIR score. Five well-established DCNNs (VGGNet, ResNet, DenseNet, EfficientNet and ViT) was implemented to predict the METAVIR score. The performance of DCNNs for five-level (F0/F1/F2/F3/F4) classification was evaluated through area under the receiver operating characteristic curve (AUC) with 95% confidential interval, accuracy, sensitivity, specificity, positive and negative likelihood ratio.

RESULTS

Similar mean AUC values were achieved for five models; VGGNet (0.96), ResNet (0.96), DenseNet (0.95), EfficientNet (0.96), and ViT (0.95). The same mean accuracy (0.94) and specificity values (0.96) were yielded for all models. In terms of sensitivity, EffcientNet achieved highest mean value (0.85) while the other models produced slightly lower values range from 0.82 to 0.84.

CONCLUSION

In this study, we demonstrated that DCNNs can classify the staging of liver fibrosis according to METAVIR score with high performance using conventional B-mode images. Among them, EfficientNET that have fewer parameters and computation cost produced highest performance. From the results, we believe that DCNNs based classification of liver fibrosis may allow fast and accurate diagnosis of liver fibrosis without needs of additional equipment for add-on test and may be powerful tool for supporting radiologists in clinical practice.

MLMSeg: A multi-view learning model for ultrasound thyroid nodule segmentation

Accurate segmentation of the thyroid gland in ultrasound images is an essential initial step in distinguishing between benign and malignant nodules, thus facilitating early diagnosis. Most existing deep learning-based methods to segment thyroid nodules are learned from only a single view or two views, which limits the performance of segmenting nodules at different scales in complex ultrasound scanning environments. To address this limitation, this study proposes a multi-view learning model, abbreviated as MLMSeg. First, a deep convolutional neural network is introduced to encode the features of the local view. Second, a multi-channel transformer module is designed to capture long-range dependency correlations of global view between different nodules. Third, there are semantic relationships of structural view between features of different layers. For example, low-level features and high-level features are endowed with hidden relationships in the feature space. To this end, a cross-layer graph convolutional module is proposed to adaptively learn the correlations of high-level and low-level features by constructing graphs across different layers. In addition, in the view fusion, a channel-aware graph attention block is devised to fuse the features from the aforementioned views for accurate segmentation of thyroid nodules. To demonstrate the effectiveness of the proposed method, extensive comparative experiments were conducted with 14 baseline methods. MLMSeg achieved higher Dice coefficients (92.10% and 83.84%) and Intersection over Union scores (86.60% and 73.52%) on two different thyroid datasets. The exceptional segmentation capability of MLMSeg for thyroid nodules can greatly assist in localizing thyroid nodules and facilitating more precise measurements of their transverse and longitudinal diameters, which is of significant clinical relevance for the diagnosis of thyroid nodules.

Real-time reliable semantic segmentation of thyroid nodules in ultrasound images

Objective.Low efficiency in medical image segmentation is a common issue that limits computer-aided diagnosis development. Due to the varying positions and sizes of nodules, it is not easy to accurately segment ultrasound images. This study aims to propose a segmentation model that maintains high efficiency while improving accuracy.Approach. We propose a novel layer that integrates the advantages of dense connectivity, dilated convolution, and factorized filters to maintain excellent efficiency while improving accuracy. Dense connectivity optimizes feature reuse, dilated convolution redesigns layers, and factorized convolution improves efficiency. Moreover, we propose a loss function optimization method from a pixel perspective to increase the network's accuracy further.Main results.Experiments on the Thyroid dataset show that our method achieves 81.70% intersection-over-union (IoU), 90.50% true positive rate (TPR), and 0.25% false positive rate (FPR). In terms of accuracy, our method outperforms the state-of-the-art methods, with twice faster inference and nearly 400 times fewer parameters. Meanwhile, in a test on an External Thyroid dataset, our method achieves 77.03% IoU, 82.10% TPR, and 0.16% FPR, demonstrating our proposed model's robustness.Significance.We propose a real-time semantic segmentation architecture for thyroid nodule segmentation in ultrasound images called fully convolution dense dilated network (FCDDN). Our method runs fast with a few parameters and is suitable for medical devices requiring real-time segmentation.

Radiomics Harmonization in Ultrasound Images for Cervical Cancer Lymph Node Metastasis Prediction Using Cycle-GAN

Background: Ultrasound (US) based radiomics is susceptible to variations in scanners, sonographers. Objective: To retrospectively investigate the feasibility of an adapted cycle generative adversarial networks (CycleGAN) in the style transfer to improve US based radiomics in the prediction of lymph node metastasis (LNM) with images from multiple scanners for patients with early cervical cancer (ECC). Methods: The CycleGAN was firstly trained to transfer paired US phantom images from one US device to another one; the model was then further trained and tested with clinical US images of ECC by transferring images from four US devices to one specific device; finally, the adapted model was tested with its effects on the radiomics feature harmonization and accuracy of LNM prediction in US based radiomics for ECC patients. Results: Phantom study demonstrated an increased radiomics harmonization using CycleGAN with an average Pearson correlation coefficient of 0.60 and 0.81 for radiomics features extracted from original and generated images in correlation with the target phantom images, respectively. Additionally, the image quality metric Peak Signal-to-Noise Ratio (PSNR) was increased from 11.18 for the original images to 15.45 for the generated image. Clinical US images of 169 ECC patients were enrolled for style transfer model training and validation. The area under curve (AUC) of LNM prediction radiomics models with features extracted from generated images of different style transfer models ranged from 0.73 to 0.85. The AUC was improved from 0.78 with features extracted from original images to 0.85 with style transferred images. Conclusions: The adapted CycleGAN network is able to increase the radiomics feature harmonization for images from different ultrasound equipment based on image domain and improve the LNM prediction accuracy for ECC.

A knowledge-interpretable multi-task learning framework for automated thyroid nodule diagnosis in ultrasound videos

Ultrasound has become the most widely used modality for thyroid nodule diagnosis, due to its portability, real-time feedback, lack of toxicity, and low cost. Recently, the computer-aided diagnosis (CAD) of thyroid nodules has attracted significant attention. However, most existing techniques can only be applied to either static images with prominent features (manually selected from scanning videos) or rely on 'black boxes' that cannot provide interpretable results. In this study, we develop a user-friendly framework for the automated diagnosis of thyroid nodules in ultrasound videos, by simulating the typical diagnostic workflow used by radiologists. This process consists of two orderly part-to-whole tasks. The first interprets the characteristics of each image using prior knowledge, to obtain corresponding frame-wise TI-RADS scores. Associated embedded representations not only provide diagnostic information for radiologists but also reduce computational costs. The second task models temporal contextual information in an embedding vector sequence and selectively enhances important information to distinguish benign and malignant thyroid nodules, thereby improving the efficiency and generalizability of the proposed framework. Experimental results demonstrated this approach outperformed other state-of-the-art video classification methods. In addition to assisting radiologists in understanding model predictions, these CAD results could further ease diagnostic workloads and improve patient care.

SDA-Net: Self-distillation driven deformable attentive aggregation network for thyroid nodule identification in ultrasound images

Early detection and accurate identification of thyroid nodules are the major challenges in controlling and treating thyroid cancer that can be difficult even for expert physicians. Currently, many computer-aided diagnosis (CAD) systems have been developed to assist this clinical process. However, most of these systems are unable to well capture geometrically diverse thyroid nodule representations from ultrasound images with subtle and various characteristic differences, resulting in suboptimal diagnosis and lack of clinical interpretability, which may affect their credibility in the clinic. In this context, a novel end-to-end network equipped with a deformable attention network and a distillation-driven interaction aggregation module (DIAM) is developed for thyroid nodule identification. The deformable attention network learns to identify discriminative features of nodules under the guidance of the deformable attention module (DAM) and an online class activation mapping (CAM) mechanism and suggests the location of diagnostic features to provide interpretable predictions. DIAM is designed to take advantage of the complementarities of adjacent layers, thus enhancing the representation capabilities of aggregated features; driven by an efficient self-distillation mechanism, the identification process is complemented with more multi-scale semantic information to calibrate the diagnosis results. Experimental results on a large dataset with varying nodule appearances show that the proposed network can achieve competitive performance in nodule diagnosis and provide interpretability suitable for clinical needs.

DefCor-Net: Physics-aware ultrasound deformation correction

The recovery of morphologically accurate anatomical images from deformed ones is challenging in ultrasound (US) image acquisition, but crucial to accurate and consistent diagnosis, particularly in the emerging field of computer-assisted diagnosis. This article presents a novel physics-aware deformation correction approach based on a coarse-to-fine, multi-scale deep neural network (DefCor-Net). To achieve pixel-wise performance, DefCor-Net incorporates biomedical knowledge by estimating pixel-wise stiffness online using a U-shaped feature extractor. The deformation field is then computed using polynomial regression by integrating the measured force applied by the US probe. Based on real-time estimation of pixel-by-pixel tissue properties, the learning-based approach enables the potential for anatomy-aware deformation correction. To demonstrate the effectiveness of the proposed DefCor-Net, images recorded at multiple locations on forearms and upper arms of six volunteers are used to train and validate DefCor-Net. The results demonstrate that DefCor-Net can significantly improve the accuracy of deformation correction to recover the original geometry (Dice Coefficient: from 14.3±20.9 to 82.6±12.1 when the force is 6N). Code:https://github.com/KarolineZhy/DefCorNet.

TS-DSANN: Texture and shape focused dual-stream attention neural network for benign-malignant diagnosis of thyroid nodules in ultrasound images

Recently, accurate diagnosis of thyroid nodules has played a critical role in precision medicine and healthcare system management. Due to complicated changes in ultrasound features of texture, and similar visual appearance of benign-malignant nodules, the identification of cancerous thyroid lesions from a given ultrasound image still faces challenges for even experienced radiologists. Learning-based computer-aided diagnosis (CAD) systems have accordingly attracted more and more attention recently. However, little research is committed to developing a deep learning-based CAD system that has greater conformity with radiologists' diagnostic decision-making. In this study, we devise a texture and shape focused dual-stream attention neural network, dubbed TS-DSANN. Specifically, in the texture focused stream, we utilize the ImageNet pre-trained ResNet34 to guide the network to recognize texture-related nodule attributes. Meanwhile, in the shape focused stream, in addition to using ResNet34 backbone, jointly learning from scratch with the contour obtained by contour detection module to enhance the extraction of shape features. Afterward, we employ a concatenation operation to aggregate the abovementioned two stream networks for capturing richer and more representative features. Finally, we further utilize an online class activation mapping mechanism to assist the dual-stream network in generating a localization heatmap to obtain more visualization attention to the nodule from the whole image, and supervise classifier's attention in decision-making. Experimental results conducted on the two-center thyroid nodule ultrasound datasets verify that our proposed method has improved the classification performance, superior to the state-of-the-art methods.

Objective assessment of segmentation models for thyroid ultrasound images

Ultrasound features related to thyroid lesions structure, shape, volume, and margins are considered to determine cancer risk. Automatic segmentation of the thyroid lesion would allow the sonographic features to be estimated. On the basis of clinical ultrasonography B-mode scans, a multi-output CNN-based semantic segmentation is used to separate thyroid nodules' cystic & solid components. Semantic segmentation is an automatic technique that labels the ultrasound (US) pixels with an appropriate class or pixel category, i.e., belongs to a lesion or background. In the present study, encoder-decoder-based semantic segmentation models i.e. SegNet using VGG16, UNet, and Hybrid-UNet implemented for segmentation of thyroid US images. For this work, 820 thyroid US images are collected from the DDTI and ultrasoundcases.info (USC) datasets. These segmentation models were trained using a transfer learning approach with original and despeckled thyroid US images. The performance of segmentation models is evaluated by analyzing the overlap region between the true contour lesion marked by the radiologist and the lesion retrieved by the segmentation model. The mean intersection of union (mIoU), mean dice coefficient (mDC) metrics, TPR, TNR, FPR, and FNR metrics are used to measure performance. Based on the exhaustive experiments and performance evaluation parameters it is observed that the proposed Hybrid-UNet segmentation model segments thyroid nodules and cystic components effectively.

Automated diagnosis and management of follicular thyroid nodules based on the devised small-dataset interpretable foreground optimization network deep learning: a multicenter diagnostic study

BACKGROUND

Currently, follicular thyroid carcinoma (FTC) has a relatively low incidence with a lack of effective preoperative diagnostic means. To reduce the need for invasive diagnostic procedures and to address information deficiencies inherent in a small dataset, we utilized interpretable foreground optimization network deep learning to develop a reliable preoperative FTC detection system.

METHODS

In this study, a deep learning model (FThyNet) was established using preoperative ultrasound images. Data on patients in the training and internal validation cohort ( n =432) were obtained from Ruijin Hospital, China. Data on patients in the external validation cohort ( n =71) were obtained from four other clinical centers. We evaluated the predictive performance of FThyNet and its ability to generalize across multiple external centers and compared the results yielded with assessments from physicians directly predicting FTC outcomes. In addition, the influence of texture information around the nodule edge on the prediction results was evaluated.

RESULTS

FThyNet had a consistently high accuracy in predicting FTC with an area under the receiver operating characteristic curve (AUC) of 89.0% [95% CI 87.0-90.9]. Particularly, the AUC for grossly invasive FTC reached 90.3%, which was significantly higher than that of the radiologists (56.1% [95% CI 51.8-60.3]). The parametric visualization study found that those nodules with blurred edges and relatively distorted surrounding textures were more likely to have FTC. Furthermore, edge texture information played an important role in FTC prediction with an AUC of 68.3% [95% CI 61.5-75.5], and highly invasive malignancies had the highest texture complexity.

CONCLUSION

FThyNet could effectively predict FTC, provide explanations consistent with pathological knowledge, and improve clinical understanding of the disease.

Assessment of encoder-decoder-based segmentation models for thyroid ultrasound images

Encoder-decoder-based semantic segmentation models classify image pixels into the corresponding class, such as the ROI (region of interest) or background. In the present study, simple / dilated convolution / series / directed acyclic graph (DAG)-based encoder-decoder semantic segmentation models have been implemented, i.e., SegNet (VGG16), SegNet (VGG19), U-Net, mobileNetv2, ResNet18, ResNet50, Xception and Inception networks for the segment TTUS(Thyroid Tumor Ultrasound) images. Transfer learning has been used to train these segmentation networks using original and despeckled TTUS images. The performance of the networks has been calculated using mIoU and mDC metrics. Based on the exhaustive experiments, it has been observed that ResNet50-based segmentation model obtained the best results objectively with values 0.87 for mIoU, 0.94 for mDC, and also according to radiologist opinion on shape, margin, and echogenicity characteristics of segmented lesions. It is noted that the segmentation model, namely ResNet50, provides better segmentation based on objective and subjective assessment. It may be used in the healthcare system to identify thyroid nodules accurately in real time.

Artificial intelligence in thyroid ultrasound

Artificial intelligence (AI), particularly deep learning (DL) algorithms, has demonstrated remarkable progress in image-recognition tasks, enabling the automatic quantitative assessment of complex medical images with increased accuracy and efficiency. AI is widely used and is becoming increasingly popular in the field of ultrasound. The rising incidence of thyroid cancer and the workload of physicians have driven the need to utilize AI to efficiently process thyroid ultrasound images. Therefore, leveraging AI in thyroid cancer ultrasound screening and diagnosis cannot only help radiologists achieve more accurate and efficient imaging diagnosis but also reduce their workload. In this paper, we aim to present a comprehensive overview of the technical knowledge of AI with a focus on traditional machine learning (ML) algorithms and DL algorithms. We will also discuss their clinical applications in the ultrasound imaging of thyroid diseases, particularly in differentiating between benign and malignant nodules and predicting cervical lymph node metastasis in thyroid cancer. Finally, we will conclude that AI technology holds great promise for improving the accuracy of thyroid disease ultrasound diagnosis and discuss the potential prospects of AI in this field.

Fine-Needle Aspiration Biopsy Evaluation-Oriented Thyroid Carcinoma Auxiliary Diagnosis

OBJECTIVE

Thyroid carcinoma is one of the most common diseases with an increasing incidence worldwide in recent years. In clinical diagnosis, medical practitioners normally take a preliminary thyroid nodule grading so that highly suspected thyroid nodules can be taken into the fine-needle aspiration (FNA) biopsy to evaluate the malignancy. However, subjective misinterpretations might lead to ambiguous risk stratification of thyroid nodules and unnecessary FNA biopsy.

METHODS

We propose a thyroid carcinoma auxiliary diagnosis method for fine-needle aspiration biopsy evaluation. Through integration of several deep learning models into a multibranch network for thyroid nodule risk stratification in the Thyroid Imaging Reporting and Data System (TIRADS) with pathological features and cascading of a discriminator, our proposed method provides an intelligent auxiliary diagnosis to assist medical practitioners in determining the necessity for further FNA.

DISCUSSION

Experimental results revealed that not only was the rate at which nodules are falsely diagnosed as malignant nodules effectively reduced, which avoids the unnecessary high cost and pain of aspiration biopsy, but also previously missing detected cases were identified with high possibility. By comparing the physicians' diagnosis alone with machine-assisted diagnosis, physicians' diagnostic performance improved with the aid of our proposed method, illustrating that our model can be very helpful in clinical practice.

CONCLUSION

Our proposed method might help medical practitioners avoid subjective interpretations and inter-observer variability. For patients, reliable diagnosis is provided and unnecessary painful diagnostics can be avoided. In other superficial organs such as metastatic lymph nodes and salivary gland tumors, the proposed method might also provide a reliable auxiliary diagnosis for risk stratification.

Intra-class consistency and inter-class discrimination feature learning for automatic skin lesion classification

Automated skin lesion classification has been proved to be capable of improving the diagnostic performance for dermoscopic images. Although many successes have been achieved, accurate classification remains challenging due to the significant intra-class variation and inter-class similarity. In this article, a deep learning method is proposed to increase the intra-class consistency as well as the inter-class discrimination of learned features in the automatic skin lesion classification. To enhance the inter-class discriminative feature learning, a CAM-based (class activation mapping) global-lesion localization module is proposed by optimizing the distance of CAMs for the same dermoscopic image generated by different skin lesion tasks. Then, a global features guided intra-class similarity learning module is proposed to generate the class center according to the deep features of all samples in one class and the history feature of one sample during the learning process. In this way, the performance can be improved with the collaboration of CAM-based inter-class feature discriminating and global features guided intra-class feature concentrating. To evaluate the effectiveness of the proposed method, extensive experiments are conducted on the ISIC-2017 and ISIC-2018 datasets. Experimental results with different backbones have demonstrated that the proposed method has good generalizability and can adaptively focus on more discriminative regions of the skin lesion.

Robust co-teaching learning with consistency-based noisy label correction for medical image classification

PURPOSE

Deep neural networks (DNNs) have made great achievements in computer-aided diagnostic systems, but the success highly depends on massive data with high-quality labels. However, for many medical image datasets, a considerable number of noisy labels are introduced by inter- and intra-observer variability, thus hampering DNNs' performance. To address this problem, a robust noisy label correction method with the co-teaching learning paradigm is proposed.

METHODS

The proposed method aims to reduce the effect of noisy labels by correcting or removing them. It consists of two modules. An adaptive noise rate estimation module is employed to calculate the dataset's noise rate, which is helpful to detect noisy labels but is usually unavailable in clinical applications. A consistency-based noisy label correction module aims to detect noisy labels and correct them to reduce the disturbance from noisy labels and exploit useful information in data.

RESULTS

Experiments are conducted on the public skin lesion dataset ISIC-2017, ISIC-2019, and our constructed thyroid ultrasound image dataset. The results demonstrate that the proposed method outperforms other noisy label learning methods in medical image classification tasks. It is also evaluated on the natural image dataset CIFAR-10 to show its generalization.

CONCLUSION

This paper proposes a noisy label correction method to handle noisy labels in medical image datasets. Experimental results show that it can self-adapt to different datasets and efficiently correct noisy labels, which is suitable for medical image classification.

Thyroid region prior guided attention for ultrasound segmentation of thyroid nodules

Ultrasound segmentation of thyroid nodules is a challenging task, which plays an vital role in the diagnosis of thyroid cancer. However, the following two factors limit the development of automatic thyroid nodule segmentation algorithms: (1) existing automatic nodule segmentation algorithms that directly apply semantic segmentation techniques can easily mistake non-thyroid areas as nodules, because of the lack of the thyroid gland region perception, the large number of similar areas in the ultrasonic images, and the inherently low contrast images; (2) the currently available dataset (i.e., DDTI) is small and collected from a single center, which violates the fact that thyroid ultrasound images are acquired from various devices in real-world situations. To overcome the lack of thyroid gland region prior knowledge, we design a thyroid region prior guided feature enhancement network (TRFE+) for accurate thyroid nodule segmentation. Specifically, (1) a novel multi-task learning framework that simultaneously learns the nodule size, gland position, and the nodule position is designed; (2) an adaptive gland region feature enhancement module is proposed to make full use of the thyroid gland prior knowledge; (3) a normalization approach with respect to the channel dimension is applied to alleviate the domain gap during the training process. To facilitate the development of thyroid nodule segmentation, we have contributed TN3K: an open-access dataset containing 3493 thyroid nodule images with high-quality nodule masks labeling from various devices and views. We perform a thorough evaluation based on the TN3K test set and DDTI to demonstrate the effectiveness of the proposed method. Code and data are available at https://github.com/haifangong/TRFE-Net-for-thyroid-nodule-segmentation.

Automatic detection and diagnosis of thyroid ultrasound images based on attention mechanism

Incidents of thyroid cancer have dramatically increased in recent years; however, early ultrasound diagnosis can reduce morbidity and mortality. The work in clinical situations relies heavily on the subjective experience of the sonographer. Numerous computer-aided diagnostic techniques exist, but most consider how good the results are, ignoring the pre-image collecting and its usefulness in post-clinical practise. To address these issues, this study proposes a computer-aided diagnosis method based on an attentional mechanism. Due to its lightweight properties, the model can rapidly identify nodules and distinguish between benign and malignant ones without using much hardware. The model uses a bounding box to locate the thyroid nodule and determines whether it is benign or cancerous, and outputs the diagnostic result of the thyroid nodule ultrasound images. The latest attention mechanisms are used to get better results at a fraction of the cost. Additionally, ultrasound images with different features of benign and malignant thyroid nodules were collected following the Thyroid Imaging Reporting and Data System standards. The experimental results showed that the approach identifies and classifies thyroid nodules rapidly and effectively; the mAP value of the results reached 0.89, and the mAP value of malignant nodules reached 0.94, with detection rate of single image reached 7 ms. Young physicians and small hospitals with limited resources can benefit from using this method to assist with thyroid ultrasound examination diagnosis.

A novel ultrasound image diagnostic method for thyroid nodules

The incidence of thyroid nodules is increasing year by year. Accurate determination of benign and malignant nodules is an important basis for formulating treatment plans. Ultrasonography is the most widely used methodology in the diagnosis of benign and malignant nodules, but diagnosis by doctors is highly subjective, and the rates of missed diagnosis and misdiagnosis are high. To improve the accuracy of clinical diagnosis, this paper proposes a new diagnostic model based on deep learning. The diagnostic model adopts the diagnostic strategy of localization-classification. First, the distribution laws of the nodule size and nodule aspect ratio are obtained through data statistics, a multiscale localization network structure is a priori designed, and the nodule aspect ratio is obtained from the positioning results. Then, uncropped ultrasound images and nodule area image are correspondingly input into a two-way classification network, and an improved attention mechanism is used to enhance the feature extraction performance. Finally, the deep features, the shallow features, and the nodule aspect ratio are fused, and a fully connected layer is used to complete the classification of benign and malignant nodules. The experimental dataset consists of 4021 ultrasound images, where each image has been labeled under the guidance of doctors, and the ratio of the training set, validation set, and test set sizes is close to 3:1:1. The experimental results show that the accuracy of the multiscale localization network reaches 93.74%, and that the accuracy, specificity, and sensitivity of the classification network reach 86.34%, 81.29%, and 90.48%, respectively. Compared with the champion model of the TNSCUI 2020 classification competition, the accuracy rate is 1.52 points higher. Therefore, the network model proposed in this paper can effectively diagnose benign and malignant thyroid nodules.

Classification for thyroid nodule using ViT with contrastive learning in ultrasound images

The lack of representative features between benign nodules, especially level 3 of Thyroid Imaging Reporting and Data System (TI-RADS), and malignant nodules limits diagnostic accuracy, leading to inconsistent interpretation, overdiagnosis, and unnecessary biopsies. We propose a Vision-Transformer-based (ViT) thyroid nodule classification model using contrast learning, called TC-ViT, to improve accuracy of diagnosis and specificity of biopsy recommendations. ViT can explore the global features of thyroid nodules well. Nodule images are used as ROI to enhance the local features of the ViT. Contrast learning can minimize the representation distance between nodules of the same category, enhance the representation consistency of global and local features, and achieve accurate diagnosis of TI-RADS 3 or malignant nodules. The test results achieve an accuracy of 86.9%. The evaluation metrics show that the network outperforms other classical deep learning-based networks in terms of classification performance. TC-ViT can achieve automatic classification of TI-RADS 3 and malignant nodules on ultrasound images. It can also be used as a key step in computer-aided diagnosis for comprehensive analysis and accurate diagnosis. The code will be available at https://github.com/Jiawei217/TC-ViT.

Symmetrical awareness network for cross-site ultrasound thyroid nodule segmentation

Recent years have seen remarkable progress of learning-based methods on Ultrasound Thyroid Nodules segmentation. However, with very limited annotations, the multi-site training data from different domains makes the task remain challenging. Due to domain shift, the existing methods cannot be well generalized to the out-of-set data, which limits the practical application of deep learning in the field of medical imaging. In this work, we propose an effective domain adaptation framework which consists of a bidirectional image translation module and two symmetrical image segmentation modules. The framework improves the generalization ability of deep neural networks in medical image segmentation. The image translation module conducts the mutual conversion between the source domain and the target domain, while the symmetrical image segmentation modules perform image segmentation tasks in both domains. Besides, we utilize adversarial constraint to further bridge the domain gap in feature space. Meanwhile, a consistency loss is also utilized to make the training process more stable and efficient. Experiments on a multi-site ultrasound thyroid nodule dataset achieve 96.22% for PA and 87.06% for DSC in average, demonstrating that our method performs competitively in cross-domain generalization ability with state-of-the-art segmentation methods.

Differential regional importance mapping for thyroid nodule malignancy prediction with potential to improve needle aspiration biopsy sampling reliability

Objective

Existing guidelines for ultrasound-guided fine-needle aspiration biopsy lack specifications on sampling sites, but the number of biopsies improves diagnostic reliability. We propose the use of class activation maps (CAMs) and our modified malignancy-specific heat maps that locate important deep representations of thyroid nodules for class predictions.

Methods

We applied adversarial noise perturbations to the segmented concentric "hot" nodular regions of equal sizes to differentiate regional importance for the malignancy diagnostic performances of an accurate ultrasound-based artificial intelligence computer-aided diagnosis (AI-CADx) system using 2,602 retrospectively collected thyroid nodules with known histopathological diagnosis.

Results

The AI system demonstrated high diagnostic performance with an area under the curve (AUC) value of 0.9302 and good nodule identification capability with a median dice coefficient >0.9 when compared to radiologists' segmentations. Experiments confirmed that the CAM-based heat maps reflect the differentiable importance of different nodular regions for an AI-CADx system to make its predictions. No less importantly, the hot regions in malignancy heat maps of ultrasound images in comparison with the inactivated regions of the same 100 malignant nodules randomly selected from the dataset had higher summed frequency-weighted feature scores of 6.04 versus 4.96 rated by radiologists with more than 15 years of ultrasound examination experience according to widely used ultrasound-based risk stratification American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS) in terms of nodule composition, echogenicity, and echogenic foci, excluding shape and margin attributes, which could only be evaluated on the whole rather than on the sub-nodular component levels. In addition, we show examples demonstrating good spatial correspondence of highlighted regions of malignancy heat map to malignant tumor cell-rich regions in hematoxylin and eosin-stained histopathological images.

Conclusion

Our proposed CAM-based ultrasonographic malignancy heat map provides quantitative visualization of malignancy heterogeneity within a tumor, and it is of clinical interest to investigate in the future its usefulness to improve fine-needle aspiration biopsy (FNAB) sampling reliability by targeting potentially more suspicious sub-nodular regions.

Automatic classification of thyroid nodules in ultrasound images using a multi-task attention network guided by clinical knowledge

Thyroid cancer has been the most prevalent cancer in the recent three decades. Ultrasonography is one of the mainly used methods for diagnosing thyroid nodules. Several computer-aided diagnostic methods were proposed to aid radiologists in analyzing ultrasound images of the thyroid gland. Most methods, however, only determine the benignity or malignancy of the thyroid nodule and do not explain the decision-making process of them, which cannot gain the trustworthiness of clinicians because they are not consistent with the physician's diagnostic process. In our work, we design a multi-task branching attention network in which each of the descriptors of the ACR TI-RADS lexicon is first classified. All respective scores are calculated to get the risk stratification of the nodule. Ultimately, based on the risk stratification, the benignity or malignancy of the nodule is determined. This work provides an automated method that incorporates the ACR TI-RADS characterization of thyroid nodules for detecting the level of risk and the benignity or malignancy of thyroid nodules. Thus the work establishes the trustworthiness of clinicians in deep learning models and improves physician efficiency and diagnostic rates to some extent compared to previous studies. For the diagnosis of thyroid nodules, evaluation indices including accuracy, sensitivity, and specificity were 93.55%, 93.86%, and 93.14%, respectively. The experiments show that our approach obtains comparable performance to most advanced methods in diagnosing ultrasound images of the thyroid nodules and is supported by explanations in clinical terms using the ACR TI-RADS lexicon.

A Soft Label Deep Learning to Assist Breast Cancer Target Therapy and Thyroid Cancer Diagnosis

According to the World Health Organization Report 2022, cancer is the most common cause of death contributing to nearly one out of six deaths worldwide. Early cancer diagnosis and prognosis have become essential in reducing the mortality rate. On the other hand, cancer detection is a challenging task in cancer pathology. Trained pathologists can detect cancer, but their decisions are subjective to high intra- and inter-observer variability, which can lead to poor patient care owing to false-positive and false-negative results. In this study, we present a soft label fully convolutional network (SL-FCN) to assist in breast cancer target therapy and thyroid cancer diagnosis, using four datasets. To aid in breast cancer target therapy, the proposed method automatically segments human epidermal growth factor receptor 2 (HER2) amplification in fluorescence in situ hybridization (FISH) and dual in situ hybridization (DISH) images. To help in thyroid cancer diagnosis, the proposed method automatically segments papillary thyroid carcinoma (PTC) on Papanicolaou-stained fine needle aspiration and thin prep whole slide images (WSIs). In the evaluation of segmentation of HER2 amplification in FISH and DISH images, we compare the proposed method with thirteen deep learning approaches, including U-Net, U-Net with InceptionV5, Ensemble of U-Net with Inception-v4, Inception-Resnet-v2 encoder, and ResNet-34 encoder, SegNet, FCN, modified FCN, YOLOv5, CPN, SOLOv2, BCNet, and DeepLabv3+ with three different backbones, including MobileNet, ResNet, and Xception, on three clinical datasets, including two DISH datasets on two different magnification levels and a FISH dataset. The result on DISH breast dataset 1 shows that the proposed method achieves high accuracy of 87.77 ± 14.97%, recall of 91.20 ± 7.72%, and F1-score of 81.67 ± 17.76%, while, on DISH breast dataset 2, the proposed method achieves high accuracy of 94.64 ± 2.23%, recall of 83.78 ± 6.42%, and F1-score of 85.14 ± 6.61% and, on the FISH breast dataset, the proposed method achieves high accuracy of 93.54 ± 5.24%, recall of 83.52 ± 13.15%, and F1-score of 86.98 ± 9.85%, respectively. Furthermore, the proposed method outperforms most of the benchmark approaches by a significant margin (p <0.001). In evaluation of segmentation of PTC on Papanicolaou-stained WSIs, the proposed method is compared with three deep learning methods, including Modified FCN, U-Net, and SegNet. The experimental result demonstrates that the proposed method achieves high accuracy of 99.99 ± 0.01%, precision of 92.02 ± 16.6%, recall of 90.90 ± 14.25%, and F1-score of 89.82 ± 14.92% and significantly outperforms the baseline methods, including U-Net and FCN (p <0.001). With the high degree of accuracy, precision, and recall, the results show that the proposed method could be used in assisting breast cancer target therapy and thyroid cancer diagnosis with faster evaluation and minimizing human judgment errors.

Multi-objective data enhancement for deep learning-based ultrasound analysis

Recently, Deep Learning based automatic generation of treatment recommendation has been attracting much attention. However, medical datasets are usually small, which may lead to over-fitting and inferior performances of deep learning models. In this paper, we propose multi-objective data enhancement method to indirectly scale up the medical data to avoid over-fitting and generate high quantity treatment recommendations. Specifically, we define a main and several auxiliary tasks on the same dataset and train a specific model for each of these tasks to learn different aspects of knowledge in limited data scale. Meanwhile, a Soft Parameter Sharing method is exploited to share learned knowledge among models. By sharing the knowledge learned by auxiliary tasks to the main task, the proposed method can take different semantic distributions into account during the training process of the main task. We collected an ultrasound dataset of thyroid nodules that contains Findings, Impressions and Treatment Recommendations labeled by professional doctors. We conducted various experiments on the dataset to validate the proposed method and justified its better performance than existing methods.

Novel Human Artificial Intelligence Hybrid Framework Pinpoints Thyroid Nodule Malignancy and Identifies Overlooked Second-Order Ultrasonographic Features

We present a Human Artificial Intelligence Hybrid (HAIbrid) integrating framework that reweights Thyroid Imaging Reporting and Data System (TIRADS) features and the malignancy score predicted by a convolutional neural network (CNN) for nodule malignancy stratification and diagnosis. We defined extra ultrasonographical features from color Doppler images to explore malignancy-relevant features. We proposed Gated Attentional Factorization Machine (GAFM) to identify second-order interacting features trained via a 10 fold distribution-balanced stratified cross-validation scheme on ultrasound images of 3002 nodules all finally characterized by postoperative pathology (1270 malignant ones), retrospectively collected from 131 hospitals. Our GAFM-HAIbrid model demonstrated significant improvements in Area Under the Curve (AUC) value (p-value < 10−5), reaching about 0.92 over the standalone CNN (~0.87) and senior radiologists (~0.86), and identified a second-order vascularity localization and morphological pattern which was overlooked if only first-order features were considered. We validated the advantages of the integration framework on an already-trained commercial CNN system and our findings using an extra set of ultrasound images of 500 nodules. Our HAIbrid framework allows natural integration to clinical workflow for thyroid nodule malignancy risk stratification and diagnosis, and the proposed GAFM-HAIbrid model may help identify novel diagnosis-relevant second-order features beyond ultrasonography.

The latest application progress of radiomics in prediction and diagnosis of liver diseases

INTRODUCTION

Early detection and individualized treatment of patients with liver disease is the key to survival. Radiomics can extract high-throughput quantitative features by multimode imaging, which has good application prospects for the diagnosis, staging and prognosis of benign and malignant liver diseases. Therefore, this paper summarizes the current research status in the field of liver disease, in order to help these patients achieve personalized and precision medical care.

AREAS COVERED

This paper uses several keywords on the PubMed database to search the references, and reviews the workflow of traditional radiomics, as well as the characteristics and influencing factors of different imaging modes. At the same time, the references on the application of imaging in different benign and malignant liver diseases were also summarized.

EXPERT OPINION

For patients with liver disease, the traditional imaging evaluation can only provide limited information. Radiomics exploits the characteristics of high-throughput and high-dimensional extraction, enabling liver imaging capabilities far beyond the scope of traditional visual image analysis. Recent studies have demonstrated the prospect of this technology in personalized diagnosis and treatment decision in various fields of the liver. However, further clinical validation is needed in its application and practice.

Ultrasound Image Classification of Thyroid Nodules Based on Deep Learning

A thyroid nodule, which is defined as abnormal growth of thyroid cells, indicates excessive iodine intake, thyroid degeneration, inflammation, and other diseases. Although thyroid nodules are always non-malignant, the malignancy likelihood of a thyroid nodule grows steadily every year. In order to reduce the burden on doctors and avoid unnecessary fine needle aspiration (FNA) and surgical resection, various studies have been done to diagnose thyroid nodules through deep-learning-based image recognition analysis. In this study, to predict the benign and malignant thyroid nodules accurately, a novel deep learning framework is proposed. Five hundred eight ultrasound images were collected from the Third Hospital of Hebei Medical University in China for model training and validation. First, a ResNet18 model, pretrained on ImageNet, was trained by an ultrasound image dataset, and a random sampling of training dataset was applied 10 times to avoid accidental errors. The results show that our model has a good performance, the average area under curve (AUC) of 10 times is 0.997, the average accuracy is 0.984, the average recall is 0.978, the average precision is 0.939, and the average F1 score is 0.957. Second, Gradient-weighted Class Activation Mapping (Grad-CAM) was proposed to highlight sensitive regions in an ultrasound image during the learning process. Grad-CAM is able to extract the sensitive regions and analyze their shape features. Based on the results, there are obvious differences between benign and malignant thyroid nodules; therefore, shape features of the sensitive regions are helpful in diagnosis to a great extent. Overall, the proposed model demonstrated the feasibility of employing deep learning and ultrasound images to estimate benign and malignant thyroid nodules.

The Application of Knowledge Distillation toward Fine-Grained Segmentation for Three-Vessel View of Fetal Heart Ultrasound Images

Objectives. Measuring anatomical parameters in fetal heart ultrasound images is crucial for the diagnosis of congenital heart disease (CHD), which is highly dependent on the clinical experience of the sonographer. To address this challenge, we propose an automated segmentation method using the channel-wise knowledge distillation technique. Methods. We design a teacher-student architecture to conduct channel-wise knowledge distillation. ROI-based cropped images and full-size images are used for the teacher and student models, respectively. It allows the student model to have both the fine-grained segmentation capability inherited from the teacher model and the ability to handle full-size test images. A total of 1,300 fetal heart ultrasound images of three-vessel view were collected and annotated by experienced doctors for training, validation, and testing. Results. We use three evaluation protocols to quantitatively evaluate the segmentation accuracy: Intersection over Union (IoU), Pixel Accuracy (PA), and Dice coefficient (Dice). We achieved better results than related methods on all evaluation metrics. In comparison with DeepLabv3+, the proposed method gets more accurate segmentation boundaries and has performance gains of 1.8% on mean IoU (66.8% to 68.6%), 2.2% on mean PA (79.2% to 81.4%), and 1.2% on mean Dice (80.1% to 81.3%). Conclusions. Our segmentation method could identify the anatomical structure in three-vessel view of fetal heart ultrasound images. Both quantitative and visual analyses show that the proposed method significantly outperforms the related methods in terms of segmentation results.

Artificial Intelligence for Thyroid Nodule Characterization: Where Are We Standing?

Simple Summary

In the present review, an up-to-date summary of the state of the art of artificial intelligence (AI) implementation for thyroid nodule characterization and cancer is provided. The opinion on the real effectiveness of AI systems remains controversial. Taking into consideration the largest and most scientifically valid studies, it is possible to state that AI provides results that are comparable or inferior to expert ultrasound specialists and radiologists. Promising data approve AI as a support tool and simultaneously highlight the need for a radiologist supervisory framework for AI provided results. Therefore, current solutions might be more suitable for educational purposes.

Abstract

Machine learning (ML) is an interdisciplinary sector in the subset of artificial intelligence (AI) that creates systems to set up logical connections using algorithms, and thus offers predictions for complex data analysis. In the present review, an up-to-date summary of the current state of the art regarding ML and AI implementation for thyroid nodule ultrasound characterization and cancer is provided, highlighting controversies over AI application as well as possible benefits of ML, such as, for example, training purposes. There is evidence that AI increases diagnostic accuracy and significantly limits inter-observer variability by using standardized mathematical algorithms. It could also be of aid in practice settings with limited sub-specialty expertise, offering a second opinion by means of radiomics and computer-assisted diagnosis. The introduction of AI represents a revolutionary event in thyroid nodule evaluation, but key issues for further implementation include integration with radiologist expertise, impact on workflow and efficiency, and performance monitoring.

A Local and Global Feature Disentangled Network: Toward Classification of Benign-Malignant Thyroid Nodules From Ultrasound Image

Thyroid nodules are one of the most common nodular lesions. The incidence of thyroid cancer has increased rapidly in the past three decades and is one of the cancers with the highest incidence. As a non-invasive imaging modality, ultrasonography can identify benign and malignant thyroid nodules, and it can be used for large-scale screening. In this study, inspired by the domain knowledge of sonographers when diagnosing ultrasound images, a local and global feature disentangled network (LoGo-Net) is proposed to classify benign and malignant thyroid nodules. This model imitates the dual-pathway structure of human vision and establishes a new feature extraction method to improve the recognition performance of nodules. We use the tissue-anatomy disentangled (TAD) block to connect the dual pathways, which decouples the cues of local and global features based on the self-attention mechanism. To verify the effectiveness of the model, we constructed a large-scale dataset and conducted extensive experiments. The results show that our method achieves an accuracy of 89.33%, which has the potential to be used in the clinical practice of doctors, including early cancer screening procedures in remote or resource-poor areas.

Multitask network for thyroid nodule diagnosis based on TI-RADS

PURPOSE

Assessment of thyroid nodules is usually relied on the experience of the radiologist and is time consuming. Classification model of thyroid nodules can not only reduce the burden on physicians but also provide objective recommendations. However, most classification models based on deep learning simply give a prediction result of the benignity or malignancy of nodules thus physicians have no way of knowing how the deep learning gets the prediction result due to the black-box nature of neural networks. In this work, we integrate the explainability directly into the outputs generated by the model through combining TI-RADS. The inference process of the proposed method is consistent with doctor's clinical diagnosis process, therefore, doctors can better explain the diagnosis results of the model to the patient.

METHODS

A multitask network based on TI-RADS (MTN-TI-RADS) for the classification of thyroid nodules is proposed. In this network, a set of TI-RADS classifications of nodules is first obtained by multitask learning, then the TI-RADS points and the corresponding risk levels are calculated, finally, nodules are classified as benign and malignant. The classification process through the network is consistent with the diagnostic process of physician, thus the results of classification can be easily understood by physicians. In addition, the attention modules are introduced to the spatial and channel domains to let the network focus more on critical features.

RESULTS

To verify the classification performance of our method, we compared the results obtained through our method with the results of the radiologist's evaluation. For the 781 test nodules in the internal dataset and the 886 test nodules in the external dataset, the sensitivity and specificity of MTN-TI-RADS were 0.988, 0.912 in internal dataset, 0.949, 0.930 in external dataset, versus the senior radiologist of 0.925 (P < 0.001), 0.816 (P = 0.005) and 0.910(P = 0.009), 0.836 (P < 0.001), respectively. And the area under the receiver operating characteristic curve (AUC) of MTN-TI-RADS was 0.981 in internal dataset, 0.973 in external dataset, versus the senior radiologist of 0.905, 0.923. For the internal dataset, we also computed the accuracy of the risk level (TR1 to TR5) and the mean absolute error (MAE). The accuracy of the risk level of the proposed method is 78%, and the MAE is 1.30. The MAE of the total points (0 to 14 points) is 1.30.

CONCLUSIONS

An effective and result-interpretable end-to-end thyroid nodule classification network (MTN-TI-RADS) is proposed. MTN-TI-RADS has superior ability to classify malignant and benign thyroid nodules compared to senior radiologists. Based on MTN-TI-RADS, a classification model with strong interpretation and a high degree of physician trust is constructed. The proposed classification network is consistent with the diagnosis process of physicians, thus is more reliable and interpretable, and has great potential for clinical application. This article is protected by copyright. All rights reserved.