Development of a machine learning-based fine-grained risk stratification system for thyroid nodules using predefined clinicoradiological features

Prediction of thyroid malignancy risk using clinical and ultrasonography features and a machine learning approach

OBJECTIVE

This study aims to develop and validate a predictive model for thyroid nodule malignancy risks using clinical and ultrasonography features and a machine learning (ML) approach.

METHODS

This retrospective study is based on the clinical and ultrasound characteristics of 1035 thyroid nodules (845 benign and 190 malignant) to develop and validate the risk prediction model. Employing multiple logistic regression, key features were selected in developing the model. Eight ML algorithms were evaluated for predicting the risks of malignancy. Finally, the predictive ability of the best-performing algorithm was compared against American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) and American Thyroid Association (ATA) guidelines.

RESULTS

Based on AUC criteria (88.3, 95% CI: 81.2-94.2), sensitivity (84.2, 95% CI: 71.1-94.7), specificity (92.3, 95% CI: 88.2-95.9), positive predictive value, (71.4, 95% CI: 60.4-83.3) and negative predictive value (96.3, 95% CI: 93.5-98.8), the XGBoost algorithm exhibited superior performance over the other ML algorithms and ACR TI-RADS and ATA. These criteria were obtained for ACR TI-RADS at 54.2%, 63.2%, 48.5%, 21.1%, and 84.8%, while for ATA, they were 44.3%, 76.3%, 27.2%, 18.4%, and 81.6%. In addition, the unnecessary fine-needle aspiration (FNA) rate with ACR TI-RADS and ATA was 43% and 63%, respectively-significantly higher than the 7% obtained with XGBoost.

CONCLUSIONS

This study demonstrated the capability of ML approaches in enhancing the accuracy of predicting thyroid malignancy risks as well as their potential benefits in optimizing healthcare resources by reducing unnecessary FNA rates. Using the proposed model through a web-based tool can facilitate clinical judgments in thyroid nodule management and personalized treatment.

KEY POINTS

Question Current risk assessment systems have limitations, with high unnecessary FNA rates compared to machine learning (ML) models. Findings The XGBoost algorithm was compared to other ML algorithms, ACR TI-RADS, and ATA and demonstrated superior performance. Clinical relevance This study demonstrated the capability of ML approaches in enhancing the accuracy of predicting thyroid malignancy. The proposed web-based tool to facilitate the prediction of thyroid nodule risk is available at https://aimedlab.ir/tnr .

[Preoperative Evaluation of Cervical Lymph Node Metastasis in Patients With Hashimoto's Thyroiditis Combined With Thyroid Papillary Carcinoma Using Machine Learning and Radiomics-Based Features: A Preliminary Study]

Objective

To analyze the radiomic and clinical features extracted from 2D ultrasound images of thyroid tumors in patients with Hashimoto's thyroiditis (HT) combined with papillary thyroid carcinoma (PTC) using machine learning (ML) models, and to explore the diagnostic performance of the method in making preoperative noninvasive identification of cervical lymph node metastasis (LNM).

Methods

A total of 528 patients with HT combined with PTC were enrolled and divided into two groups based on their pathological results of the presence or absence of LNM. The groups were subsequently designated the With LNM Group and the Without LNM Group. Three ultrasound doctors independently delineated the regions of interest and extracted radiomic features. Two modes, radiomic features and radiomics-clinical features, were used to construct random forest (RF), support vector machine (SVM), LightGBM, K-nearest neighbor (KNN), and XGBoost models. The performance of these five ML models in the two modes was evaluated by the receiver operating characteristic (ROC) curves on the test dataset, and SHapley Additive exPlanations (SHAP) was used for model visualization.

Results

All five ML models showed good performance, with area under the ROC curve (AUC) ranging from 0.798 to 0.921. LightGBM and XGBoost demonstrated the best performance, outperforming the other models (P<0.05). The ML models constructed with radiomics-clinical features performed better than those constructed using only radiomic features (P<0.05). The SHAP visualization of the best-performing models indicated that the anteroposterior diameter, superoinferior diameter, original_shape_VoxelVolume, age, wavelet-LHL_firstorder_10Percentile, and left-to-right diameter had the most significant effect on the LightGBM model. On the other hand, the superoinferior diameter, anteroposterior diameter, left-to-right diameter, original_shape_VoxelVolume, original_firstorder_InterquartileRange, and age had the most significant effect on the XGBoost model.

Conclusion

ML models based on radiomics and clinical features can accurately evaluate the cervical lymph node status in patients with HT combined with PTC. Among the 5 ML models, LightGBM and XGBoost demonstrate the best evaluation performance.

Artificial Intelligence Model Assisting Thyroid Nodule Diagnosis and Management: A Multicenter Diagnostic Study

CONTEXT

It is not clear how to integrate artificial intelligence (AI)-based models into diagnostic workflows.

OBJECTIVE

To develop and validate a deep-learning-based AI model (AI-Thyroid) for thyroid cancer diagnosis, and to explore how this improves diagnostic performance.

METHODS

The system was trained using 19 711 images of 6163 patients in a tertiary hospital (Ajou University Medical Center; AUMC). It was validated using 11 185 images of 4820 patients in 24 hospitals (test set 1) and 4490 images of 2367 patients in AUMC (test set 2). The clinical implications were determined by comparing the findings of six physicians with different levels of experience (group 1: 4 trainees, and group 2: 2 faculty radiologists) before and after AI-Thyroid assistance.

RESULTS

The area under the receiver operating characteristic (AUROC) curve of AI-Thyroid was 0.939. The AUROC, sensitivity, and specificity were 0.922, 87.0%, and 81.5% for test set 1 and 0.938, 89.9%, and 81.6% for test set 2. The AUROCs of AI-Thyroid did not differ significantly according to the prevalence of malignancies (>15.0% vs ≤15.0%, P = .226). In the simulated scenario, AI-Thyroid assistance changed the AUROC, sensitivity, and specificity from 0.854 to 0.945, from 84.2% to 92.7%, and from 72.9% to 86.6% (all P < .001) in group 1, and from 0.914 to 0.939 (P = .022), from 78.6% to 85.5% (P = .053) and from 91.9% to 92.5% (P = .683) in group 2. The interobserver agreement improved from moderate to substantial in both groups.

CONCLUSION

AI-Thyroid can improve diagnostic performance and interobserver agreement in thyroid cancer diagnosis, especially in less-experienced physicians.