A Local and Global Feature Disentangled Network: Toward Classification of Benign-Malignant Thyroid Nodules From Ultrasound Image

A Dual-Branch Cross-Modality-Attention Network for Thyroid Nodule Diagnosis Based on Ultrasound Images and Contrast-Enhanced Ultrasound Videos

Contrast-enhanced ultrasound (CEUS) has been extensively employed as an imaging modality in thyroid nodule diagnosis due to its capacity to visualise the distribution and circulation of micro-vessels in organs and lesions in a non-invasive manner. However, current CEUS-based thyroid nodule diagnosis methods suffered from: 1) the blurred spatial boundaries between nodules and other anatomies in CEUS videos, and 2) the insufficient representations of the local structural information of nodule tissues by the features extracted only from CEUS videos. In this paper, we propose a novel dual-branch network with a cross-modality-attention mechanism for thyroid nodule diagnosis by integrating the information from tow related modalities, i.e., CEUS videos and ultrasound image. The mechanism has two parts: US-attention-from-CEUS transformer (UAC-T) and CEUS-attention-from-US transformer (CAU-T). As such, this network imitates the manner of human radiologists by decomposing the diagnosis into two correlated tasks: 1) the spatio-temporal features extracted from CEUS are hierarchically embedded into the spatial features extracted from US with UAC-T for the nodule segmentation; 2) the US spatial features are used to guide the extraction of the CEUS spatio-temporal features with CAU-T for the nodule classification. The two tasks are intertwined in the dual-branch end-to-end network and optimized with the multi-task learning (MTL) strategy. The proposed method is evaluated on our collected thyroid US-CEUS dataset. Experimental results show that our method achieves the classification accuracy of 86.92%, specificity of 66.41%, and sensitivity of 97.01%, outperforming the state-of-the-art methods. As a general contribution in the field of multi-modality diagnosis of diseases, the proposed method has provided an effective way to combine static information with its related dynamic information, improving the quality of deep learning based diagnosis with an additional benefit of explainability.

DP-CLAM: A weakly supervised benign-malignant classification study based on dual-angle scanning ultrasound images of thyroid nodules

In this paper, a two-stage task weakly supervised learning algorithm is proposed. It accurately achieved patient-level classification task of benign and malignant thyroid nodules based on ultrasound images from two scanning angles: long axis and short axis of the thyroid site. In the first stage, 68,208 ultrasound scanning images of 588 patients are used to train the underlying classification model. In the second stage, feature vectors of ultrasound images with dual scan angles are extracted using the classification model in the first stage. Then the feature vectors are assigned to position sequences in the order of visual reception by the physician. Finally, the location decision is made through a weakly supervised learning approach. Combined with the dual-angle difference information carried in the overall features, our method accurately achieved benign and malignant classification of thyroid nodules at the patient level. An accuracy of 93.81 % for benign and malignant classification of patients was obtained in our test set. The accuracy of benign and malignant classification of patients with thyroid nodules is improved by our weakly supervised learning method based on a two-stage classification task. It also reduced the pressure of imaging physicians in diagnosing a large number of images. In the clinical auxiliary diagnosis, it provides an effective reference for the timely determination of thyroid nodule patients.

An unsupervised automatic texture classification method for ultrasound images of thyroid nodules

Objective.Ultrasound is the predominant modality in medical practice for evaluating thyroid nodules. Currently, diagnosis is typically based on textural information. This study aims to develop an automated texture classification approach to aid physicians in interpreting ultrasound images of thyroid nodules. However, there is currently a scarcity of pixel-level labeled datasets for the texture classes of thyroid nodules. The labeling of such datasets relies on professional and experienced doctors, requiring a significant amount of manpower. Therefore, the objective of this study is to develop an unsupervised method for classifying nodule textures.Approach.Firstly, we develop a spatial mapping network to transform the one-dimensional pixel value space into a high-dimensional space to extract comprehensive feature information. Subsequently, we outline the principles of feature selection that are suitable for clustering. Then we propose a pixel-level clustering algorithm with a region growth pattern, and a distance evaluation method for texture sets among different nodules is established.Main results.Our algorithm achieves a pixel-level classification accuracy of 0.931 for the cystic and solid region, 0.870 for the hypoechoic region, 0.959 for the isoechoic region, and 0.961 for the hyperechoic region. The efficacy of our algorithm and its concordance with human observation have been demonstrated. Furthermore, we conduct calculations and visualize the distribution of different textures in benign and malignant nodules.Significance.This method can be used for the automatic generation of pixel-level labels of thyroid nodule texture, aiding in the construction of texture datasets, and offering image analysis information for medical professionals.

Targeting CHEK1: Ginsenosides-Rh2 and Cu2O@G-Rh2 nanoparticles in thyroid cancer

Thyroid cancer (THCA) is an increasingly common malignant tumor of the endocrine system, with its incidence rising steadily in recent years. For patients who experience recurrence or metastasis, treatment options are relatively limited, and the prognosis is poor. Therefore, exploring new therapeutic strategies has become particularly urgent. This study confirmed that effective suppression of THCA cell proliferation and stimulation of apoptosis can be achieved through the application of Ginsenosides-Rh2. Through network pharmacology screening, the molecular target of Ginsenosides-Rh2 in THCA was identified as CHEK1, and its inhibitory effect was confirmed by downregulating CHEK1 protein expression. Furthermore, demonstrations conducted both in vitro and in vivo showcased that delivering Ginsenosides-Rh2 using nanoparticle carriers significantly reduced cell viability by approximately 50%, regulated DNA damage levels, apoptosis-related protein expression, and cell cycle control. The IC50 of the nanoparticle formulation was determined (B-CPAP IC50 = 88.24 μM), TPC IC50 = 79.52 μM). This study confirmed that Cu2O@G-Rh2 is effective in suppressing tumors and exhibits a significant inhibitory effect on tumor recurrence and metastasis while maintaining good safety. Cu2O@G-Rh2 nanoparticles possess excellent stability and anti-tumor efficacy. This research offers new perspectives for the treatment of THCA and demonstrates potential clinical applications.

Attention-Guided Learning With Feature Reconstruction for Skin Lesion Diagnosis Using Clinical and Ultrasound Images

Skin lesion is one of the most common diseases, and most categories are highly similar in morphology and appearance. Deep learning models effectively reduce the variability between classes and within classes, and improve diagnostic accuracy. However, the existing multi-modal methods are only limited to the surface information of lesions in skin clinical and dermatoscopic modalities, which hinders the further improvement of skin lesion diagnostic accuracy. This requires us to further study the depth information of lesions in skin ultrasound. In this paper, we propose a novel skin lesion diagnosis network, which combines clinical and ultrasound modalities to fuse the surface and depth information of the lesion to improve diagnostic accuracy. Specifically, we propose an attention-guided learning (AL) module that fuses clinical and ultrasound modalities from both local and global perspectives to enhance feature representation. The AL module consists of two parts, attention-guided local learning (ALL) computes the intra-modality and inter-modality correlations to fuse multi-scale information, which makes the network focus on the local information of each modality, and attention-guided global learning (AGL) fuses global information to further enhance the feature representation. In addition, we propose a feature reconstruction learning (FRL) strategy which encourages the network to extract more discriminative features and corrects the focus of the network to enhance the model's robustness and certainty. We conduct extensive experiments and the results confirm the superiority of our proposed method. Our code is available at: https://github.com/XCL-hub/AGFnet.

SPGAN Optimized by Piranha Foraging Optimization for Thyroid Nodule Classification in Ultrasound Images

In this research work, Semantic-Preserved Generative Adversarial Network optimized by Piranha Foraging Optimization for Thyroid Nodule Classification in Ultrasound Images (SPGAN-PFO-TNC-UI) is proposed. Initially, ultrasound images are gathered from the DDTI dataset. Then the input image is sent to the pre-processing step. During pre-processing stage, the Multi-Window Savitzky-Golay Filter (MWSGF) is employed to reduce the noise and improve the quality of the ultrasound (US) images. The pre-processed output is supplied to the Generalized Intuitionistic Fuzzy C-Means Clustering (GIFCMC). Here, the ultrasound image's Region of Interest (ROI) is segmented. The segmentation output is supplied to the Fully Numerical Laplace Transform (FNLT) to extract the features, such as geometric features like solidity, orientation, roundness, main axis length, minor axis length, bounding box, convex area, and morphological features, like area, perimeter, aspect ratio, and AP ratio. The Semantic-Preserved Generative Adversarial Network (SPGAN) separates the image as benign or malignant nodules. Generally, SPGAN does not express any optimization adaptation methodologies for determining the best parameters to ensure the accurate classification of thyroid nodules. Therefore, the Piranha Foraging Optimization (PFO) algorithm is proposed to improve the SPGAN classifier and accurately identify the thyroid nodules. The metrics, like F-score, accuracy, error rate, precision, sensitivity, specificity, ROC, computing time is examined. The proposed SPGAN-PFO-TNC-UI method attains 30.54%, 21.30%, 27.40%, and 18.92% higher precision and 26.97%, 20.41%, 15.09%, and 18.27% lower error rate compared with existing techniques, like Thyroid detection and classification using DNN with Hybrid Meta-Heuristic and LSTM (TD-DL-HMH-LSTM), Quantum-Inspired convolutional neural networks for optimized thyroid nodule categorization (QCNN-OTNC), Thyroid nodules classification under Follow the Regularized Leader Optimization based Deep Neural Networks (CTN-FRL-DNN), Automatic classification of ultrasound thyroids images using vision transformers and generative adversarial networks (ACUTI-VT-GAN) respectively.

Cross-Domain Mutual-Assistance Learning Framework for Fully Automated Diagnosis of Primary Tumor in Nasopharyngeal Carcinoma

Accurate T-staging of nasopharyngeal carcinoma (NPC) holds paramount importance in guiding treatment decisions and prognosticating outcomes for distinct risk groups. Regrettably, the landscape of deep learning-based techniques for T-staging in NPC remains sparse, and existing methodologies often exhibit suboptimal performance due to their neglect of crucial domain-specific knowledge pertinent to primary tumor diagnosis. To address these issues, we propose a new cross-domain mutual-assistance learning framework for fully automated diagnosis of primary tumor using H&N MR images. Specifically, we tackle primary tumor diagnosis task with the convolutional neural network consisting of a 3D cross-domain knowledge perception network (CKP net) for excavated cross-domain-invariant features emphasizing tumor intensity variations and internal tumor heterogeneity, and a multi-domain mutual-information sharing fusion network (M2SF net), comprising a dual-pathway domain-specific representation module and a mutual information fusion module, for intelligently gauging and amalgamating multi-domain, multi-scale T-stage diagnosis-oriented features. The proposed 3D cross-domain mutual-assistance learning framework not only embraces task-specific multi-domain diagnostic knowledge but also automates the entire process of primary tumor diagnosis. We evaluate our model on an internal and an external MR images dataset in a three-fold cross-validation paradigm. Exhaustive experimental results demonstrate that our method outperforms the other algorithms, and obtains promising performance for tumor segmentation and T-staging. These findings underscore its potential for clinical application, offering valuable assistance to clinicians in treatment decision-making and prognostication for various risk groups.

Do as Sonographers Think: Contrast-Enhanced Ultrasound for Thyroid Nodules Diagnosis via Microvascular Infiltrative Awareness

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

Fast inter-frame motion correction in contrast-free ultrasound quantitative microvasculature imaging using deep learning

Contrast-free ultrasound quantitative microvasculature imaging shows promise in several applications, including the assessment of benign and malignant lesions. However, motion represents one of the major challenges in imaging tumor microvessels in organs that are prone to physiological motions. This study aims at addressing potential microvessel image degradation in in vivo human thyroid due to its proximity to carotid artery. The pulsation of the carotid artery induces inter-frame motion that significantly degrades microvasculature images, resulting in diagnostic errors. The main objective of this study is to reduce inter-frame motion artifacts in high-frame-rate ultrasound imaging to achieve a more accurate visualization of tumor microvessel features. We propose a low-complex deep learning network comprising depth-wise separable convolutional layers and hybrid adaptive and squeeze-and-excite attention mechanisms to correct inter-frame motion in high-frame-rate images. Rigorous validation using phantom and in-vivo data with simulated inter-frame motion indicates average improvements of 35% in Pearson correlation coefficients (PCCs) between motion corrected and reference data with respect to that of motion corrupted data. Further, reconstruction of microvasculature images using motion-corrected frames demonstrates PCC improvement from 31 to 35%. Another thorough validation using in-vivo thyroid data with physiological inter-frame motion demonstrates average improvement of 20% in PCC and 40% in mean inter-frame correlation. Finally, comparison with the conventional image registration method indicates the suitability of proposed network for real-time inter-frame motion correction with 5000 times reduction in motion corrected frame prediction latency.

An interpretable two-branch bi-coordinate network based on multi-grained domain knowledge for classification of thyroid nodules in ultrasound images

Computer-aided diagnosis (CAD) for thyroid nodules has been studied for years, yet there are still reliability and interpretability challenges due to the lack of clinically-relevant evidence. To address this issue, inspired by Thyroid Imaging Reporting and Data System (TI-RADS), we propose a novel interpretable two-branch bi-coordinate network based on multi-grained domain knowledge. First, we transform the two types of domain knowledge provided by TI-RADS, namely region-based and boundary-based knowledge, into labels at multi-grained levels: coarse-grained classification labels, and fine-grained region segmentation masks and boundary localization vectors. We combine these two labels to form the Multi-grained Domain Knowledge Representation (MG-DKR) of TI-RADS. Then we design a Two-branch Bi-coordinate network (TB2C-net) which utilizes two branches to predict MG-DKR from both Cartesian and polar images, and uses an attention-based integration module to integrate the features of the two branches for benign-malignant classification. We validated our method on a large cohort containing 3245 patients (with 3558 nodules and 6466 ultrasound images). Results show that our method achieves competitive performance with AUC of 0.93 and ACC of 0.87 compared with other state-of-the-art methods. Ablation experiment results demonstrate the effectiveness of the TB2C-net and MG-DKR, and the knowledge attention map from the integration module provides the interpretability for benign-malignant classification.

A multi-slice attention fusion and multi-view personalized fusion lightweight network for Alzheimer's disease diagnosis

OBJECTIVE

Alzheimer's disease (AD) is a type of neurological illness that significantly impacts individuals' daily lives. In the intelligent diagnosis of AD, 3D networks require larger computational resources and storage space for training the models, leading to increased model complexity and training time. On the other hand, 2D slices analysis may overlook the 3D structural information of MRI and can result in information loss.

APPROACH

We propose a multi-slice attention fusion and multi-view personalized fusion lightweight network for automated AD diagnosis. It incorporates a multi-branch lightweight backbone to extract features from sagittal, axial, and coronal view of MRI, respectively. In addition, we introduce a novel multi-slice attention fusion module, which utilizes a combination of global and local channel attention mechanism to ensure consistent classification across multiple slices. Additionally, a multi-view personalized fusion module is tailored to assign appropriate weights to the three views, taking into account the varying significance of each view in achieving accurate classification results. To enhance the performance of the multi-view personalized fusion module, we utilize a label consistency loss to guide the model's learning process. This encourages the acquisition of more consistent and stable representations across all three views.

MAIN RESULTS

The suggested strategy is efficient in lowering the number of parameters and FLOPs, with only 3.75M and 4.45G respectively, and accuracy improved by 10.5% to 14% in three tasks. Moreover, in the classification tasks of AD vs. CN, AD vs. MCI and MCI vs. CN, the accuracy of the proposed method is 95.63%, 86.88% and 85.00%, respectively, which is superior to the existing methods.

CONCLUSIONS

The results show that the proposed approach not only excels in resource utilization, but also significantly outperforms the four comparison methods in terms of accuracy and sensitivity, particularly in detecting early-stage AD lesions. It can precisely capture and accurately identify subtle brain lesions, providing crucial technical support for early intervention and treatment.

Sub-features orthogonal decoupling: Detecting bone wall absence via a small number of abnormal examples for temporal CT images

The absence of bone wall located in the jugular bulb and sigmoid sinus of the temporal bone is one of the important reasons for pulsatile tinnitus. Automatic and accurate detection of these abnormal singes in CT slices has important theoretical significance and clinical value. Due to the shortage of abnormal samples, imbalanced samples, small inter-class differences, and low interpretability, existing deep-learning methods are greatly challenged. In this paper, we proposed a sub-features orthogonal decoupling model, which can effectively disentangle the representation features into class-specific sub-features and class-independent sub-features in a latent space. The former contains the discriminative information, while, the latter preserves information for image reconstruction. In addition, the proposed method can generate image samples using category conversion by combining the different class-specific sub-features and the class-independent sub-features, achieving corresponding mapping between deep features and images of specific classes. The proposed model improves the interpretability of the deep model and provides image synthesis methods for downstream tasks. The effectiveness of the method was verified in the detection of bone wall absence in the temporal bone jugular bulb and sigmoid sinus.

A transformer-based multi-task deep learning model for simultaneous T-stage identification and segmentation of nasopharyngeal carcinoma

Background

Accurate tumor target contouring and T staging are vital for precision radiation therapy in nasopharyngeal carcinoma (NPC). Identifying T-stage and contouring the Gross tumor volume (GTV) manually is a laborious and highly time-consuming process. Previous deep learning-based studies have mainly been focused on tumor segmentation, and few studies have specifically addressed the tumor staging of NPC.

Objectives

To bridge this gap, we aim to devise a model that can simultaneously identify T-stage and perform accurate segmentation of GTV in NPC.

Materials and methods

We have developed a transformer-based multi-task deep learning model that can perform two tasks simultaneously: delineating the tumor contour and identifying T-stage. Our retrospective study involved contrast-enhanced T1-weighted images (CE-T1WI) of 320 NPC patients (T-stage: T1-T4) collected between 2017 and 2020 at our institution, which were randomly allocated into three cohorts for three-fold cross-validations, and conducted the external validation using an independent test set. We evaluated the predictive performance using the area under the receiver operating characteristic curve (ROC-AUC) and accuracy (ACC), with a 95% confidence interval (CI), and the contouring performance using the Dice similarity coefficient (DSC) and average surface distance (ASD).

Results

Our multi-task model exhibited sound performance in GTV contouring (median DSC: 0.74; ASD: 0.97 mm) and T staging (AUC: 0.85, 95% CI: 0.82-0.87) across 320 patients. In early T category tumors, the model achieved a median DSC of 0.74 and ASD of 0.98 mm, while in advanced T category tumors, it reached a median DSC of 0.74 and ASD of 0.96 mm. The accuracy of automated T staging was 76% (126 of 166) for early stages (T1-T2) and 64% (99 of 154) for advanced stages (T3-T4). Moreover, experimental results show that our multi-task model outperformed the other single-task models.

Conclusions

This study emphasized the potential of multi-task model for simultaneously delineating the tumor contour and identifying T-stage. The multi-task model harnesses the synergy between these interrelated learning tasks, leading to improvements in the performance of both tasks. The performance demonstrates the potential of our work for delineating the tumor contour and identifying T-stage and suggests that it can be a practical tool for supporting clinical precision radiation therapy.

Neural harmony: revolutionizing thyroid nodule diagnosis with hybrid networks and genetic algorithms

In the contemporary world, thyroid disease poses a prevalent health issue, particularly affecting women's well-being. Recognizing the significance of maternal thyroid (MT) hormones in fetal neurodevelopment during the first half of pregnancy, this study introduces the HNN-GSO model. This groundbreaking hybrid approach, utilizing the MT dataset, integrates ResNet-50 and Artificial Neural Network (ANN) within a Glow-worm Swarm Optimization (GSO) framework for optimal parameter tuning. With a comprehensive methodology involving dataset preprocessing and Genetic Algorithm (GA) for feature selection, our model leverages ResNet-50 for feature extraction and ANN for classification tasks. Implemented in Python, the HNN-GSO model outperforms existing models, including K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF), ResNet, GoogleNet, and ANN, achieving an impressive accuracy rate of 98%. This success underscores the effectiveness of our approach in complex classification tasks within machine learning (ML) and pattern recognition, specifically tailored to the Thyroid Ultrasound Images (TUI) Dataset. To provide a comprehensive understanding of performance, additional statistical measures such as precision, recall, and F1 score were considered. The HNN-GSO model consistently outperformed competitors across these metrics, showcasing its superiority in MT classification. The HNN-GSO model seamlessly combines ResNet-50's feature extraction, ANN's classification robustness, and GSO's optimization for unparalleled performance. This research offers a promising framework for advancing ML methodologies, enhancing accuracy, and efficiency in classification tasks related to MT health.

Thyroid Ultrasound Image Database and Marker Mask Inpainting Method for Research and Development

OBJECTIVE

The main objective of this study was to build a rich and high-quality thyroid ultrasound image database (TUD) for computer-aided diagnosis (CAD) systems to support accurate diagnosis and prognostic modeling of thyroid disorders. Because most of the raw thyroid ultrasound images contain artificial markers, which seriously affect the robustness of CAD systems because of their strong prior location information, we propose a marker mask inpainting (MMI) method to erase artificial markers and improve image quality.

METHODS

First, a set of thyroid ultrasound images were collected from the General Hospital of the Northern Theater Command. Then, two modules were designed in MMI, namely, the marker detection (MD) module and marker erasure (ME) module. The MD module detects all markers in the image and stores them in a binary mask. According to the binary mask, the ME module erases the markers and generates an unmarked image. Finally, a new TUD based on the marked images and unmarked images was built. The TUD is carefully annotated and statistically analyzed by professional physicians to ensure accuracy and consistency. Moreover, several normal thyroid gland images and some ancillary information on benign and malignant nodules are provided.

RESULTS

Several typical segmentation models were evaluated on the TUD. The experimental results revealed that our TUD can facilitate the development of more accurate CAD systems for the analysis of thyroid nodule-related lesions in ultrasound images. The effectiveness of our MMI method was determined in quantitative experiments.

CONCLUSION

The rich and high-quality resource TUD promotes the development of more effective diagnostic and treatment methods for thyroid diseases. Furthermore, MMI for erasing artificial markers and generating unmarked images is proposed to improve the quality of thyroid ultrasound images. Our TUD database is available at https://github.com/NEU-LX/TUD-Datebase.

Is Attention all You Need in Medical Image Analysis? A Review

Medical imaging is a key component in clinical diagnosis, treatment planning and clinical trial design, accounting for almost 90% of all healthcare data. CNNs achieved performance gains in medical image analysis (MIA) over the last years. CNNs can efficiently model local pixel interactions and be trained on small-scale MI data. Despite their important advances, typical CNN have relatively limited capabilities in modelling "global" pixel interactions, which restricts their generalisation ability to understand out-of-distribution data with different "global" information. The recent progress of Artificial Intelligence gave rise to Transformers, which can learn global relationships from data. However, full Transformer models need to be trained on large-scale data and involve tremendous computational complexity. Attention and Transformer compartments ("Transf/Attention") which can well maintain properties for modelling global relationships, have been proposed as lighter alternatives of full Transformers. Recently, there is an increasing trend to co-pollinate complementary local-global properties from CNN and Transf/Attention architectures, which led to a new era of hybrid models. The past years have witnessed substantial growth in hybrid CNN-Transf/Attention models across diverse MIA problems. In this systematic review, we survey existing hybrid CNN-Transf/Attention models, review and unravel key architectural designs, analyse breakthroughs, and evaluate current and future opportunities as well as challenges. We also introduced an analysis framework on generalisation opportunities of scientific and clinical impact, based on which new data-driven domain generalisation and adaptation methods can be stimulated.

PST-Radiomics: a PET/CT lymphoma classification method based on pseudo spatial-temporal radiomic features and structured atrous recurrent convolutional neural network

Objective.Existing radiomic methods tend to treat each isolated tumor as an inseparable whole, when extracting radiomic features. However, they may discard the critical intra-tumor metabolic heterogeneity (ITMH) information, that contributes to triggering tumor subtypes. To improve lymphoma classification performance, we propose a pseudo spatial-temporal radiomic method (PST-Radiomics) based on positron emission tomography computed tomography (PET/CT).Approach.Specifically, to enable exploitation of ITMH, we first present a multi-threshold gross tumor volume sequence (GTVS). Next, we extract 1D radiomic features based on PET images and each volume in GTVS and create a pseudo spatial-temporal feature sequence (PSTFS) tightly interwoven with ITMH. Then, we reshape PSTFS to create 2D pseudo spatial-temporal feature maps (PSTFM), of which the columns are elements of PSTFS. Finally, to learn from PSTFM in an end-to-end manner, we build a light-weighted pseudo spatial-temporal radiomic network (PSTR-Net), in which a structured atrous recurrent convolutional neural network serves as a PET branch to better exploit the strong local dependencies in PSTFM, and a residual convolutional neural network is used as a CT branch to exploit conventional radiomic features extracted from CT volumes.Main results.We validate PST-Radiomics based on a PET/CT lymphoma subtype classification task. Experimental results quantitatively demonstrate the superiority of PST-Radiomics, when compared to existing radiomic methods.Significance.Feature map visualization of our method shows that it performs complex feature selection while extracting hierarchical feature maps, which qualitatively demonstrates its superiority.

SDA-Net: Self-distillation driven deformable attentive aggregation network for thyroid nodule identification in ultrasound images

Early detection and accurate identification of thyroid nodules are the major challenges in controlling and treating thyroid cancer that can be difficult even for expert physicians. Currently, many computer-aided diagnosis (CAD) systems have been developed to assist this clinical process. However, most of these systems are unable to well capture geometrically diverse thyroid nodule representations from ultrasound images with subtle and various characteristic differences, resulting in suboptimal diagnosis and lack of clinical interpretability, which may affect their credibility in the clinic. In this context, a novel end-to-end network equipped with a deformable attention network and a distillation-driven interaction aggregation module (DIAM) is developed for thyroid nodule identification. The deformable attention network learns to identify discriminative features of nodules under the guidance of the deformable attention module (DAM) and an online class activation mapping (CAM) mechanism and suggests the location of diagnostic features to provide interpretable predictions. DIAM is designed to take advantage of the complementarities of adjacent layers, thus enhancing the representation capabilities of aggregated features; driven by an efficient self-distillation mechanism, the identification process is complemented with more multi-scale semantic information to calibrate the diagnosis results. Experimental results on a large dataset with varying nodule appearances show that the proposed network can achieve competitive performance in nodule diagnosis and provide interpretability suitable for clinical needs.

Radiomics-based ultrasound models for thyroid nodule differentiation in Hashimoto's thyroiditis

Background

Previous models for differentiating benign and malignant thyroid nodules(TN) have predominantly focused on the characteristics of the nodules themselves, without considering the specific features of the thyroid gland(TG) in patients with Hashimoto's thyroiditis(HT). In this study, we analyzed the clinical and ultrasound radiomics(USR) features of TN in patients with HT and constructed a model for differentiating benign and malignant nodules specifically in this population.

Methods

We retrospectively collected clinical and ultrasound data from 227 patients with TN and concomitant HT(161 for training, 66 for testing). Two experienced sonographers delineated the TG and TN regions, and USR features were extracted using Python. Lasso regression and logistic analysis were employed to select relevant USR features and clinical data to construct the model for differentiating benign and malignant TN. The performance of the model was evaluated using area under the curve(AUC), calibration curves, and decision curve analysis(DCA).

Results

A total of 1,162 USR features were extracted from TN and the TG in the 227 patients with HT. Lasso regression identified 14 features, which were used to construct the TN score, TG score, and TN+TG score. Univariate analysis identified six clinical predictors: TI-RADS, echoic type, aspect ratio, boundary, calcification, and thyroid function. Multivariable analysis revealed that incorporating USR scores improved the performance of the model for differentiating benign and malignant TN in patients with HT. Specifically, the TN+TG score resulted in the highest increase in AUC(from 0.83 to 0.94) in the clinical prediction model. Calibration curves and DCA demonstrated higher accuracy and net benefit for the TN+TG+clinical model.

Conclusion

USR features of both the TG and TN can be utilized for differentiating benign and malignant TN in patients with HT. These findings highlight the importance of considering the entire TG in the evaluation of TN in HT patients, providing valuable insights for clinical decision-making in this population.

Osteoporosis prediction in lumbar spine X-ray images using the multi-scale weighted fusion contextual transformer network

Osteoporosis is a bone-related disease characterized by decreased bone density and mass, leading to brittle fractures. Osteoporosis assessment from radiographs using a deep learning algorithm has proven a low-cost alternative to the golden standard DXA. Due to the considerable noise and low contrast, automated diagnosis of osteoporosis in X-ray images still poses a significant challenge for traditional diagnostic methods. In this paper, an end-to-end transformer-style network was proposed, termed FCoTNet, to overcome the shortcoming of insufficient fusion of texture information and local features in the traditional CoTNet. To extract complementary geometric representations at each scale of the transformer module, we integrated parallel multi-scale feature extraction architectures in each unit layer of FCoTNet to utilize convolution to aggregate features from different receptive fields. Moreover, in order to extract small-scale texture features which were more critical to the diagnosis of osteoporosis in radiographs, larger fusion weights were assigned to the feature maps with small-size receptive fields. Afterward, the multi-scale global modeling was conducted by self-attention mechanism. The proposed model was first investigated on a private lumbar spine X-ray dataset with the 5-fold cross-validation strategy, obtaining an average accuracy of 78.29 ± 0.93 %, an average sensitivity of 69.72 ± 2.35 %, and an average specificity of 88.92 ± 0.67 % for the multi-classification of normal, osteopenia, and osteoporosis categories. We then conducted a controlled trial with five orthopedic clinicians to evaluate the clinical value of the model. The average clinician's accuracy improved from 61.50 ± 10.79 % unaided to 80.00 ± 5.92 % aided (18.50 % improvement), sensitivity improved from 64.38 ± 8.07 % unaided to 83.31 ± 5.43 % aided (18.93 % improvement), and specificity improved from 80.11 ± 4.72 % unaided to 89.94 ± 3.82 % aided (9.83 % improvement). Meanwhile, the prediction consistency among clinicians significantly improved with the assistance of FCoTNet. Furthermore, the proposed model showed good robustness on an external test dataset. These investigations indicate that the proposed deep learning model achieves state-of-the-art performance for osteoporosis prediction, which substantially improves osteoporosis screening and reduced osteoporosis fractures.

TUSPM-NET: A multi-task model for thyroid ultrasound standard plane recognition and detection of key anatomical structures of the thyroid

The thyroid gland is a vital gland located in the anterior part of the neck. Ultrasound imaging of the thyroid gland is a non-invasive and widely used technique for diagnosing nodular growth, inflammation, and enlargement of the thyroid gland. In ultrasonography, the acquisition of ultrasound standard planes is crucial for disease diagnosis. However, the acquisition of standard planes in ultrasound examinations can be subjective, laborious and heavily reliant on the sonographer's clinical experience. To overcome these challenges, we design a multi-task model TUSP Multi-task Network (TUSPM-NET) that can recognize Thyroid Ultrasound Standard Plane (TUSP) and detect key anatomical structures in TUSPs in real-time. To improve TUSPM-NET's accuracy and learn prior knowledge in medical images, we proposed the plane target classes loss function and the plane targets position filter. Additionally, we collected 9778 TUSP images of 8 standard planes to train and validate the model. Experiments have shown that TUSPM-NET can accurately detect anatomical structures in TUSPs and recognize TUSP images. Compared to current models with better performance, TUSPM-NET's object detection map@0.5:0.95 improves by 9.3%; the precision and recall of plane recognition improve by 3.49% and 4.39%, respectively. Furthermore, TUSPM-NET recognizes and detects a TUSP image in just 19.9 ms, which means that the method is well suited to the needs of real-time clinical scanning.

CDNet: Contrastive Disentangled Network for Fine-Grained Image Categorization of Ocular B-Scan Ultrasound

Precise and rapid categorization of images in the B-scan ultrasound modality is vital for diagnosing ocular diseases. Nevertheless, distinguishing various diseases in ultrasound still challenges experienced ophthalmologists. Thus a novel contrastive disentangled network (CDNet) is developed in this work, aiming to tackle the fine-grained image categorization (FGIC) challenges of ocular abnormalities in ultrasound images, including intraocular tumor (IOT), retinal detachment (RD), posterior scleral staphyloma (PSS), and vitreous hemorrhage (VH). Three essential components of CDNet are the weakly-supervised lesion localization module (WSLL), contrastive multi-zoom (CMZ) strategy, and hyperspherical contrastive disentangled loss (HCD-Loss), respectively. These components facilitate feature disentanglement for fine-grained recognition in both the input and output aspects. The proposed CDNet is validated on our ZJU Ocular Ultrasound Dataset (ZJUOUSD), consisting of 5213 samples. Furthermore, the generalization ability of CDNet is validated on two public and widely-used chest X-ray FGIC benchmarks. Quantitative and qualitative results demonstrate the efficacy of our proposed CDNet, which achieves state-of-the-art performance in the FGIC task.

Multimodality US versus Thyroid Imaging Reporting and Data System Criteria in Recommending Fine-Needle Aspiration of Thyroid Nodules

Background Current guidelines recommend the use of conventional US for risk stratification and management of thyroid nodules. However, fine-needle aspiration (FNA) is often recommended in benign nodules. Purpose To compare the diagnostic performance of multimodality US (including conventional US, strain elastography, and contrast-enhanced US [CEUS]) with the American College of Radiology Thyroid Imaging Reporting and Data System (TI-RADS) in the recommendation of FNA for thyroid nodules to reduce unnecessary biopsies. Materials and Methods In this prospective study, 445 consecutive participants with thyroid nodules from nine tertiary referral hospitals were recruited between October 2020 and May 2021. With univariable and multivariable logistic regression, the prediction models incorporating sonographic features, evaluated with interobserver agreement, were constructed and internally validated with bootstrap resampling technique. In addition, discrimination, calibration, and decision curve analysis were performed. Results A total of 434 thyroid nodules confirmed at pathologic analysis (259 malignant thyroid nodules) in 434 participants (mean age, 45 years ± 12 [SD]; 307 female participants) were included. Four multivariable models incorporated participant age, nodule features at US (proportion of cystic components, echogenicity, margin, shape, punctate echogenic foci), elastography features (stiffness), and CEUS features (blood volume). In recommending FNA in thyroid nodules, the highest area under the receiver operating characteristic curve (AUC) was 0.85 (95% CI: 0.81, 0.89) for the multimodality US model, and the lowest AUC was 0.63 (95% CI: 0.59, 0.68) for TI-RADS (P < .001). At the 50% risk threshold, 31% (95% CI: 26, 38) of FNA procedures could be avoided with multimodality US compared with 15% (95% CI: 12, 19) with TI-RADS (P < .001). Conclusion Multimodality US had better performance in recommending FNA to avoid unnecessary biopsies than the TI-RADS. Clinical trial registration no. NCT04574258 © RSNA, 2023 Supplemental material is available for this article.