1
|
Zou LL, Zhang Q, Yao Z, He Y, Zhou J. Integrating artificial intelligence (S-Detect software) and contrast-enhanced ultrasound for enhanced diagnosis of thyroid nodules: A comprehensive evaluation study. JOURNAL OF CLINICAL ULTRASOUND : JCU 2024. [PMID: 39235299 DOI: 10.1002/jcu.23810] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/12/2024] [Accepted: 08/18/2024] [Indexed: 09/06/2024]
Abstract
PURPOSE This study aims to assess the diagnostic efficacy of Korean Thyroid imaging reporting and data system (K-TIRADS), S-Detect software and contrast-enhanced ultrasound (CEUS) when employed individually, as well as their combined application, for the evaluation of thyroid nodules, with the objective of identifying the optimal method for diagnosing thyroid nodules. METHODS Two hundred and sixty eight cases pathologically proven of thyroid nodules were retrospectively enrolled. Each nodule was classified according to K-TIRADS. S-Detect software was utilized for intelligent analysis. CEUS was employed to acquire contrast-enhanced features. RESULTS The area under curve (AUC) values for diagnosing benign and malignant thyroid nodules using K-TIRADS alone, S-Detect software alone, CEUS alone, the combined application of K-TIRADS and CEUS, the combined application of S-Detect software and CEUS were 0.668, 0.668, 0.719, 0.741, and 0.759, respectively (p < 0.001). The sensitivity rate of S-Detect software was 89.9% (p < 0.001). It was the highest of the five diagnostic methods above. CONCLUSION The utilization of S-Detect software can be served as a powerful tool for early screening. Notably, the combined utilization of S-Detect software with CEUS demonstrates superior diagnostic performance compared to employing K-TIRADS, S-Detect software, CEUS used individually, as well as the combined application of K-TIRADS with CEUS.
Collapse
Affiliation(s)
- Lu-Lu Zou
- Department of Ultrasound, Yichang Central People's Hospital (First Clinical Medical College of Three Gorges University), Yichang, Hubei, China
| | - Qi Zhang
- Department of Ultrasound, Yichang Central People's Hospital (First Clinical Medical College of Three Gorges University), Yichang, Hubei, China
| | - Zhi Yao
- Department of Ultrasound, Yichang Central People's Hospital (First Clinical Medical College of Three Gorges University), Yichang, Hubei, China
| | - Yong He
- Department of Ultrasound, Yichang Central People's Hospital (First Clinical Medical College of Three Gorges University), Yichang, Hubei, China
| | - Jun Zhou
- Department of Ultrasound, Yichang Second People's Hospital (Second Clinical Medical College of Three Gorges University), Yichang, Hubei, China
| |
Collapse
|
2
|
Sant VR, Radhachandran A, Ivezic V, Lee DT, Livhits MJ, Wu JX, Masamed R, Arnold CW, Yeh MW, Speier W. From Bench-to-Bedside: How Artificial Intelligence is Changing Thyroid Nodule Diagnostics, a Systematic Review. J Clin Endocrinol Metab 2024; 109:1684-1693. [PMID: 38679750 PMCID: PMC11180510 DOI: 10.1210/clinem/dgae277] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/30/2024] [Revised: 04/04/2024] [Accepted: 04/16/2024] [Indexed: 05/01/2024]
Abstract
CONTEXT Use of artificial intelligence (AI) to predict clinical outcomes in thyroid nodule diagnostics has grown exponentially over the past decade. The greatest challenge is in understanding the best model to apply to one's own patient population, and how to operationalize such a model in practice. EVIDENCE ACQUISITION A literature search of PubMed and IEEE Xplore was conducted for English-language publications between January 1, 2015 and January 1, 2023, studying diagnostic tests on suspected thyroid nodules that used AI. We excluded articles without prospective or external validation, nonprimary literature, duplicates, focused on nonnodular thyroid conditions, not using AI, and those incidentally using AI in support of an experimental diagnostic outside standard clinical practice. Quality was graded by Oxford level of evidence. EVIDENCE SYNTHESIS A total of 61 studies were identified; all performed external validation, 16 studies were prospective, and 33 compared a model to physician prediction of ground truth. Statistical validation was reported in 50 papers. A diagnostic pipeline was abstracted, yielding 5 high-level outcomes: (1) nodule localization, (2) ultrasound (US) risk score, (3) molecular status, (4) malignancy, and (5) long-term prognosis. Seven prospective studies validated a single commercial AI; strengths included automating nodule feature assessment from US and assisting the physician in predicting malignancy risk, while weaknesses included automated margin prediction and interobserver variability. CONCLUSION Models predominantly used US images to predict malignancy. Of 4 Food and Drug Administration-approved products, only S-Detect was extensively validated. Implementing an AI model locally requires data sanitization and revalidation to ensure appropriate clinical performance.
Collapse
Affiliation(s)
- Vivek R Sant
- Division of Endocrine Surgery, UT Southwestern Medical Center, Dallas, TX 75390, USA
| | - Ashwath Radhachandran
- Biomedical Artificial Intelligence Research Lab, UCLA Department of Bioengineering, Los Angeles, CA 90024, USA
| | - Vedrana Ivezic
- Biomedical Artificial Intelligence Research Lab, UCLA Department of Bioengineering, Los Angeles, CA 90024, USA
| | - Denise T Lee
- Department of Surgery, Icahn School of Medicine at Mount Sinai Hospital, New York, NY 10029, USA
| | - Masha J Livhits
- Section of Endocrine Surgery, UCLA David Geffen School of Medicine, Los Angeles, CA 90095, USA
| | - James X Wu
- Section of Endocrine Surgery, UCLA David Geffen School of Medicine, Los Angeles, CA 90095, USA
| | - Rinat Masamed
- Department of Radiology, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Corey W Arnold
- Biomedical Artificial Intelligence Research Lab, UCLA Department of Bioengineering, Los Angeles, CA 90024, USA
| | - Michael W Yeh
- Section of Endocrine Surgery, UCLA David Geffen School of Medicine, Los Angeles, CA 90095, USA
| | - William Speier
- Biomedical Artificial Intelligence Research Lab, UCLA Department of Bioengineering, Los Angeles, CA 90024, USA
| |
Collapse
|
3
|
Xia M, Song F, Zhao Y, Xie Y, Wen Y, Zhou P. Ultrasonography-based radiomics and computer-aided diagnosis in thyroid nodule management: performance comparison and clinical strategy optimization. Front Endocrinol (Lausanne) 2023; 14:1140816. [PMID: 37251675 PMCID: PMC10213653 DOI: 10.3389/fendo.2023.1140816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/09/2023] [Accepted: 05/01/2023] [Indexed: 05/31/2023] Open
Abstract
Objectives To compare ultrasonography (US) feature-based radiomics and computer-aided diagnosis (CAD) models for predicting malignancy in thyroid nodules, and to evaluate their utility for thyroid nodule management. Methods This prospective study included 262 thyroid nodules obtained between January 2022 and June 2022. All nodules previously underwent standardized US image acquisition, and the nature of the nodules was confirmed by the pathological results. The CAD model exploited two vertical US images of the thyroid nodule to differentiate the lesions. The least absolute shrinkage and operator algorithm (LASSO) was applied to choose radiomics features with excellent predictive properties for building a radiomics model. Ultimately, the area under the receiver operating characteristic curve (AUC) and calibration curves were assessed to compare diagnostic performance between the models. DeLong's test was used to analyze the difference between groups. Both models were used to revise the American College of Radiology Thyroid Imaging Reporting and Data Systems (ACR TI-RADS) to provide biopsy recommendations, and their performance was compared with the original recommendations. Results Of the 262 thyroid nodules, 157 were malignant, and the remaining 105 were benign. The diagnostic performance of radiomics, CAD, and ACR TI-RADS models had an AUC of 0.915 (95% confidence interval (CI): 0.881-0.947), 0.814 (95% CI: 0.766-0.863), and 0.849 (95% CI: 0.804-0.894), respectively. DeLong's test showed a statistically significant between the AUC values of models (p < 0.05). Calibration curves showed good agreement in each model. When both models were applied to revise the ACR TI-RADS, our recommendations significantly improved the performance. The revised recommendations based on radiomics and CAD showed an increased sensitivity, accuracy, positive predictive value, and negative predictive value, and decreased unnecessary fine-needle aspiration rates. Furthermore, the radiomics model's improvement scale was more pronounced (33.3-16.7% vs. 33.3-9.7%). Conclusion The radiomics strategy and CAD system showed good diagnostic performance for discriminating thyroid nodules and could be used to optimize the ACR TI-RADS recommendation, which successfully reduces unnecessary biopsies, especially in the radiomics model.
Collapse
Affiliation(s)
- Mengwen Xia
- Department of Ultrasonography, The Third Xiangya Hospital of Central South University, Changsha, China
| | - Fulong Song
- Department of Radiology, The Third Xiangya Hospital of Central South University, Changsha, China
| | - Yongfeng Zhao
- Department of Ultrasonography, The Third Xiangya Hospital of Central South University, Changsha, China
| | - Yongzhi Xie
- Department of Radiology, The Third Xiangya Hospital of Central South University, Changsha, China
| | - Yafei Wen
- Department of Ultrasonography, The Third Xiangya Hospital of Central South University, Changsha, China
| | - Ping Zhou
- Department of Ultrasonography, The Third Xiangya Hospital of Central South University, Changsha, China
| |
Collapse
|
4
|
Cao CL, Li QL, Tong J, Shi LN, Li WX, Xu Y, Cheng J, Du TT, Li J, Cui XW. Artificial intelligence in thyroid ultrasound. Front Oncol 2023; 13:1060702. [PMID: 37251934 PMCID: PMC10213248 DOI: 10.3389/fonc.2023.1060702] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Accepted: 04/07/2023] [Indexed: 05/31/2023] Open
Abstract
Artificial intelligence (AI), particularly deep learning (DL) algorithms, has demonstrated remarkable progress in image-recognition tasks, enabling the automatic quantitative assessment of complex medical images with increased accuracy and efficiency. AI is widely used and is becoming increasingly popular in the field of ultrasound. The rising incidence of thyroid cancer and the workload of physicians have driven the need to utilize AI to efficiently process thyroid ultrasound images. Therefore, leveraging AI in thyroid cancer ultrasound screening and diagnosis cannot only help radiologists achieve more accurate and efficient imaging diagnosis but also reduce their workload. In this paper, we aim to present a comprehensive overview of the technical knowledge of AI with a focus on traditional machine learning (ML) algorithms and DL algorithms. We will also discuss their clinical applications in the ultrasound imaging of thyroid diseases, particularly in differentiating between benign and malignant nodules and predicting cervical lymph node metastasis in thyroid cancer. Finally, we will conclude that AI technology holds great promise for improving the accuracy of thyroid disease ultrasound diagnosis and discuss the potential prospects of AI in this field.
Collapse
Affiliation(s)
- Chun-Li Cao
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
- NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China
| | - Qiao-Li Li
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
- NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China
| | - Jin Tong
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
| | - Li-Nan Shi
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
- NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China
| | - Wen-Xiao Li
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
- NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China
| | - Ya Xu
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
| | - Jing Cheng
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
| | - Ting-Ting Du
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
| | - Jun Li
- Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China
- NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China
| | - Xin-Wu Cui
- Department of Medical Ultrasound, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei, China
| |
Collapse
|
5
|
Rho M, Chun SH, Lee E, Lee HS, Yoon JH, Park VY, Han K, Kwak JY. Diagnosis of thyroid micronodules on ultrasound using a deep convolutional neural network. Sci Rep 2023; 13:7231. [PMID: 37142760 PMCID: PMC10160046 DOI: 10.1038/s41598-023-34459-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Accepted: 04/30/2023] [Indexed: 05/06/2023] Open
Abstract
To assess the performance of deep convolutional neural network (CNN) to discriminate malignant and benign thyroid nodules < 10 mm in size and compare the diagnostic performance of CNN with those of radiologists. Computer-aided diagnosis was implemented with CNN and trained using ultrasound (US) images of 13,560 nodules ≥ 10 mm in size. Between March 2016 and February 2018, US images of nodules < 10 mm were retrospectively collected at the same institution. All nodules were confirmed as malignant or benign from aspirate cytology or surgical histology. Diagnostic performances of CNN and radiologists were assessed and compared for area under curve (AUC), sensitivity, specificity, accuracy, positive predictive value, and negative predictive value. Subgroup analyses were performed based on nodule size with a cut-off value of 5 mm. Categorization performances of CNN and radiologists were also compared. A total of 370 nodules from 362 consecutive patients were assessed. CNN showed higher negative predictive value (35.3% vs. 22.6%, P = 0.048) and AUC (0.66 vs. 0.57, P = 0.04) than radiologists. CNN also showed better categorization performance than radiologists. In the subgroup of nodules ≤ 5 mm, CNN showed higher AUC (0.63 vs. 0.51, P = 0.08) and specificity (68.2% vs. 9.1%, P < 0.001) than radiologists. Convolutional neural network trained with thyroid nodules ≥ 10 mm in size showed overall better diagnostic performance than radiologists in the diagnosis and categorization of thyroid nodules < 10 mm, especially in nodules ≤ 5 mm.
Collapse
Affiliation(s)
- Miribi Rho
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea
| | - Sei Hyun Chun
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea
| | - Eunjung Lee
- School of Mathematics and Computing, Yonsei University, Seoul, Korea
| | - Hye Sun Lee
- Biostatistics Collaboration Unit, Yonsei University College of Medicine, Seoul, Korea
| | - Jung Hyun Yoon
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea
| | - Vivian Youngjean Park
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea
| | - Kyunghwa Han
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea
| | - Jin Young Kwak
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea.
| |
Collapse
|
6
|
Ng CKC. Diagnostic Performance of Artificial Intelligence-Based Computer-Aided Detection and Diagnosis in Pediatric Radiology: A Systematic Review. CHILDREN (BASEL, SWITZERLAND) 2023; 10:children10030525. [PMID: 36980083 PMCID: PMC10047006 DOI: 10.3390/children10030525] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 02/13/2023] [Accepted: 03/07/2023] [Indexed: 03/30/2023]
Abstract
Artificial intelligence (AI)-based computer-aided detection and diagnosis (CAD) is an important research area in radiology. However, only two narrative reviews about general uses of AI in pediatric radiology and AI-based CAD in pediatric chest imaging have been published yet. The purpose of this systematic review is to investigate the AI-based CAD applications in pediatric radiology, their diagnostic performances and methods for their performance evaluation. A literature search with the use of electronic databases was conducted on 11 January 2023. Twenty-three articles that met the selection criteria were included. This review shows that the AI-based CAD could be applied in pediatric brain, respiratory, musculoskeletal, urologic and cardiac imaging, and especially for pneumonia detection. Most of the studies (93.3%, 14/15; 77.8%, 14/18; 73.3%, 11/15; 80.0%, 8/10; 66.6%, 2/3; 84.2%, 16/19; 80.0%, 8/10) reported model performances of at least 0.83 (area under receiver operating characteristic curve), 0.84 (sensitivity), 0.80 (specificity), 0.89 (positive predictive value), 0.63 (negative predictive value), 0.87 (accuracy), and 0.82 (F1 score), respectively. However, a range of methodological weaknesses (especially a lack of model external validation) are found in the included studies. In the future, more AI-based CAD studies in pediatric radiology with robust methodology should be conducted for convincing clinical centers to adopt CAD and realizing its benefits in a wider context.
Collapse
Affiliation(s)
- Curtise K C Ng
- Curtin Medical School, Curtin University, GPO Box U1987, Perth, WA 6845, Australia
- Curtin Health Innovation Research Institute (CHIRI), Faculty of Health Sciences, Curtin University, GPO Box U1987, Perth, WA 6845, Australia
| |
Collapse
|
7
|
Liu T, Wu C, Wang G, Jia Y, Zhu Y, Nie F. Clinical Value of Artificial Intelligence-Based Computer-Aided Diagnosis System Versus Contrast-Enhanced Ultrasound for Differentiation of Benign From Malignant Thyroid Nodules in Different Backgrounds. JOURNAL OF ULTRASOUND IN MEDICINE : OFFICIAL JOURNAL OF THE AMERICAN INSTITUTE OF ULTRASOUND IN MEDICINE 2023. [PMID: 36794594 DOI: 10.1002/jum.16195] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Revised: 01/29/2023] [Accepted: 01/30/2023] [Indexed: 06/18/2023]
Abstract
OBJECTIVES The aim of this study was to compare the value of AI-SONIC ultrasound-assisted diagnosis system versus contrast-enhanced ultrasound (CEUS) for differential diagnosis of thyroid nodules in diffuse and non-diffuse backgrounds. METHODS A total of 555 thyroid nodules with pathologically confirmed diagnosis were included in this retrospective study. The diagnostic efficacies of AI-SONIC and CEUS for differentiating benign from malignant nodules in diffuse and non-diffuse backgrounds were evaluated, with pathological diagnosis as the gold standard. RESULTS The agreement between AI-SONIC diagnosis and pathological diagnosis was moderate in diffuse backgrounds (κ = 0.417) and almost perfect in non-diffuse backgrounds (κ = 0.81). The agreement between CEUS diagnosis and pathological diagnosis was substantial in diffuse backgrounds (κ = 0.684) and moderate in non-diffuse backgrounds (κ = 0.407). In diffuse backgrounds, AI-SONIC had slightly higher sensitivity (95.7 vs 89.4%, P = .375), but CEUS had significantly higher specificity (80.0 vs 40.0%, P = .008). In non-diffuse background, AI-SONIC had significantly higher sensitivity (96.2 vs 73.4%, P < .001), specificity (82.9 vs 71.2%, P = .007), and negative predictive value (90.3 vs 53.3%, P < .001). CONCLUSION In non-diffuse backgrounds, AI-SONIC is superior to CEUS for differentiating malignant from benign thyroid nodules. In diffuse backgrounds, AI-SONIC could be useful for screening of cases to detect suspicious nodules requiring further examination by CEUS.
Collapse
Affiliation(s)
- Ting Liu
- Ultrasound Medicine Center, Lanzhou University Second Hospital, Lanzhou, China
| | - Chuang Wu
- Department of Magnetic Resonance, Lanzhou University Second Hospital, Lanzhou, China
| | - Guojuan Wang
- Ultrasound Medicine Center, Lanzhou University Second Hospital, Lanzhou, China
| | - Yingying Jia
- Ultrasound Medicine Center, Lanzhou University Second Hospital, Lanzhou, China
| | - Yangyang Zhu
- Ultrasound Medicine Center, Lanzhou University Second Hospital, Lanzhou, China
| | - Fang Nie
- Ultrasound Medicine Center, Lanzhou University Second Hospital, Lanzhou, China
| |
Collapse
|
8
|
Zhou L, Zheng LL, Zhang CJ, Wei HF, Xu LL, Zhang MR, Li Q, He GF, Ghamor-Amegavi EP, Li SY. Comparison of S-Detect and thyroid imaging reporting and data system classifications in the diagnosis of cytologically indeterminate thyroid nodules. Front Endocrinol (Lausanne) 2023; 14:1098031. [PMID: 36761203 PMCID: PMC9902707 DOI: 10.3389/fendo.2023.1098031] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/14/2022] [Accepted: 01/11/2023] [Indexed: 01/26/2023] Open
Abstract
Purpose The aim of this study was to investigate the value of S-Detect for predicting the malignant risk of cytologically indeterminate thyroid nodules (CITNs). Methods The preoperative prediction of 159 CITNs (Bethesda III, IV and V) were performed using S-Detect, Thyroid Imaging Reporting and Data System of American College of Radiology (ACR TI-RADS) and Chinese TI-RADS (C-TIRADS). First, Linear-by-Linear Association test and chi-square test were used to analyze the malignant risk of CITNs. McNemar's test and receiver operating characteristic curve were used to compare the diagnostic efficacy of S-Detect and the two TI-RADS classifications for CITNs. In addition, the McNemar's test was used to compare the diagnostic accuracy of the above three methods for different pathological types of nodules. Results The maximum diameter of the benign nodules was significantly larger than that of malignant nodules [0.88(0.57-1.42) vs 0.57(0.46-0.81), P=0.002]. The risk of malignant CITNs in Bethesda system and the two TI-RADS classifications increased with grade (all P for trend<0.001). In all the enrolled CITNs, the diagnostic results of S-Detect were significantly different from those of ACR TI-RADS and C-TIRADS, respectively (P=0.021 and P=0.007). The sensitivity and accuracy of S-Detect [95.9%(90.1%-98.5%) and 88.1%(81.7%-92.5%)] were higher than those of ACR TI-RADS [87.6%(80.1%-92.7%) and 81.8%(74.7%-87.3%)] (P=0.006 and P=0.021) and C-TIRADS [84.3%(76.3%-90.0%) and 78.6%(71.3%-84.5%)] (P=0.001 and P=0.001). Moreover, the negative predictive value and the area under curve value of S-Detect [82.8% (63.5%-93.5%) and 0.795%(0.724%-0.855%)] was higher than that of C-TIRADS [54.8%(38.8%-69.8%) and 0.724%(0.648%-0.792%] (P=0.024 and P=0.035). However, the specificity and positive predictive value of S-Detect were similar to those of ACR TI-RADS (P=1.000 and P=0.154) and C-TIRADS (P=1.000 and P=0.072). There was no significant difference in all the evaluated indicators between ACR TI-RADS and C-TIRADS (all P>0.05). The diagnostic accuracy of S-Detect (97.4%) for papillary thyroid carcinoma (PTC) was higher than that of ACR TI-RADS (90.4%) and C-TIRADS (87.8%) (P=0.021 and P=0.003). Conclusion The diagnostic performance of S-Detect in differentiating CITNs was similar to ACR TI-RADS and superior to C-TIRADS, especially for PTC.
Collapse
Affiliation(s)
- Ling Zhou
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Lin-lin Zheng
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Chuan-ju Zhang
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Hong-fen Wei
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Li-long Xu
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Mu-rui Zhang
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Qiang Li
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | - Gao-fei He
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| | | | - Shi-yan Li
- Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China
| |
Collapse
|
9
|
The Use of Artificial Intelligence in the Diagnosis and Classification of Thyroid Nodules: An Update. Cancers (Basel) 2023; 15:cancers15030708. [PMID: 36765671 PMCID: PMC9913834 DOI: 10.3390/cancers15030708] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2022] [Revised: 01/20/2023] [Accepted: 01/20/2023] [Indexed: 01/27/2023] Open
Abstract
The incidence of thyroid nodules diagnosed is increasing every year, leading to a greater risk of unnecessary procedures being performed or wrong diagnoses being made. In our paper, we present the latest knowledge on the use of artificial intelligence in diagnosing and classifying thyroid nodules. We particularly focus on the usefulness of artificial intelligence in ultrasonography for the diagnosis and characterization of pathology, as these are the two most developed fields. In our search of the latest innovations, we reviewed only the latest publications of specific types published from 2018 to 2022. We analyzed 930 papers in total, from which we selected 33 that were the most relevant to the topic of our work. In conclusion, there is great scope for the use of artificial intelligence in future thyroid nodule classification and diagnosis. In addition to the most typical uses of artificial intelligence in cancer differentiation, we identified several other novel applications of artificial intelligence during our review.
Collapse
|
10
|
Clinical value of artificial intelligence in thyroid ultrasound: a prospective study from the real world. Eur Radiol 2023:10.1007/s00330-022-09378-y. [PMID: 36622410 DOI: 10.1007/s00330-022-09378-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Revised: 12/07/2022] [Accepted: 12/13/2022] [Indexed: 01/10/2023]
Abstract
OBJECTIVE To evaluate the diagnostic performance of a commercial artificial intelligence (AI)-assisted ultrasonography (US) for thyroid nodules and to validate its value in real-world medical practice. MATERIALS AND METHODS From March 2021 to July 2021, 236 consecutive patients with 312 suspicious thyroid nodules were prospectively enrolled in this study. One experienced radiologist performed US examinations with a real-time AI system (S-Detect). US images and AI reports of the nodules were recorded. Nine residents and three senior radiologists were invited to make a "benign" or "malignant" diagnosis based on recorded US images without knowing the AI reports. After referring to AI reports, the diagnosis was made again. The diagnostic performance of AI, residents, and senior radiologists with and without AI reports were analyzed. RESULTS The sensitivity, accuracy, and AUC of the AI system were 0.95, 0.84, and 0.753, respectively, and were not statistically different from those of the experienced radiologists, but were superior to those of the residents (all p < 0.01). The AI-assisted resident strategy significantly improved the accuracy and sensitivity for nodules ≤ 1.5 cm (all p < 0.01), while reducing the unnecessary biopsy rate by up to 27.7% for nodules > 1.5 cm (p = 0.034). CONCLUSIONS The AI system achieved performance, for cancer diagnosis, comparable to that of an average senior thyroid radiologist. The AI-assisted strategy can significantly improve the overall diagnostic performance for less-experienced radiologists, while increasing the discovery of thyroid cancer ≤ 1.5 cm and reducing unnecessary biopsies for nodules > 1.5 cm in real-world medical practice. KEY POINTS • The AI system reached a senior radiologist-like level in the evaluation of thyroid cancer and could significantly improve the overall diagnostic performance of residents. • The AI-assisted strategy significantly improved ≤ 1.5 cm thyroid cancer screening AUC, accuracy, and sensitivity of the residents, leading to an increased detection of thyroid cancer while maintaining a comparable specificity to that of radiologists alone. • The AI-assisted strategy significantly reduced the unnecessary biopsy rate for thyroid nodules > 1.5 cm by the residents, while maintaining a comparable sensitivity to that of radiologists alone.
Collapse
|
11
|
Ha EJ, Lee JH, Lee DH, Na DG, Kim JH. Development of a machine learning-based fine-grained risk stratification system for thyroid nodules using predefined clinicoradiological features. Eur Radiol 2023; 33:3211-3221. [PMID: 36600122 DOI: 10.1007/s00330-022-09376-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2022] [Revised: 09/07/2022] [Accepted: 12/11/2022] [Indexed: 01/05/2023]
Abstract
OBJECTIVE We constructed and validated a machine learning-based malignancy risk estimation model using predefined clinicoradiological features, and evaluated its clinical utility for the management of thyroid nodules. METHODS In total, 5708 benign (n = 4597) and malignant (n = 1111) thyroid nodules were collected from 5081 consecutive patients treated in 26 institutions. Seventeen experienced radiologists evaluated nodule characteristics on ultrasonographic images. Eight predictive models were used to stratify the thyroid nodules according to malignancy risk; model performance was assessed via nested 10-fold cross-validation. The best-performing algorithm was externally validated using data for 454 thyroid nodules from a tertiary hospital, then compared to the Thyroid Imaging Reporting and Data System (TIRADS)-based interpretations of radiologists (American College of Radiology, European and Korean TIRADS, and AACE/ACE/AME guidelines). RESULTS The area under the receiver operating characteristic (AUROC) curves of the algorithms ranged from 0.773 to 0.862. The sensitivities, specificities, positive predictive values, and negative predictive values of the best-performing models were 74.1-76.6%, 80.9-83.4%, 49.2-51.9%, and 93.0-93.5%, respectively. For the external validation set, the ElasticNet values were 83.2%, 89.2%, 81.8%, and 90.1%, respectively. The corresponding TIRADS values were 66.5-85.0%, 61.3-80.8%, 45.9-72.1%, and 81.5-90.3%, respectively. The new model exhibited a significantly higher AUROC and specificity than did the TIRADS risk stratification, although its sensitivity was similar. CONCLUSION We developed a reliable machine learning-based predictive model that demonstrated enhanced specificity when stratifying thyroid nodules according to malignancy risk. This system will contribute to improved personalized management of thyroid nodules. KEY POINTS • The area under the receiver operating characteristic (AUROC) curve, sensitivity, and specificity of our model were 0.914, 83.2%, and 89.2%, respectively (derived using the validation dataset). • Compared to the TIRADS values, the AUROC and specificity are significantly higher, while the sensitivity is similar. • An interactive version of our AI algorithm is at http://tirads.cdss.co.kr .
Collapse
Affiliation(s)
- Eun Ju Ha
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 16499, South Korea
| | - Jeong Hoon Lee
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 16499, South Korea
| | - Da Hyun Lee
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 16499, South Korea
| | - Dong Gyu Na
- Department of Radiology, GangNeung Asan Hospital, University of Ulsan College of Medicine, Gangneung-si, Gangwon-do, 25440, South Korea
| | - Ji-Hoon Kim
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, South Korea.
| |
Collapse
|
12
|
Huang P, Zheng B, Li M, Xu L, Rabbani S, Mayet AM, Chen C, Zhan B, Jun H. The Diagnostic Value of Artificial Intelligence Ultrasound S-Detect Technology for Thyroid Nodules. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:3656572. [PMID: 36471665 PMCID: PMC9719421 DOI: 10.1155/2022/3656572] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/06/2022] [Revised: 09/15/2022] [Accepted: 09/20/2022] [Indexed: 09/19/2023]
Abstract
This study aimed to evaluate the consistency of ultrasound TI-RADS classification used by sonographers with different ultrasound diagnosis experience in the diagnosis of thyroid nodules and the diagnostic value of using artificial intelligence ultrasound S-Detect technology in the differentiation of benign and malignant thyroid lesions. 100 patients who underwent ultrasound examination of thyroid masses in our hospital from June 2019 to June 2021 and were further punctured or operated on were included in the study. Pathological results were used as the gold standard to evaluate ultrasound S-Detect technology and the value of TI-RADS classification and the combined application of the two in diagnosing benign and malignant thyroid TI-RADS 4 types of nodules, and the consistency of judgments of doctors of different ages is assessed by a Kappa value. There were 128 nodules in 100 patients, 51 benign nodules, and 77 malignant nodules. For senior physicians, the sensitivity of diagnosis using TI-RADS classification combined with ultrasound S-Detect technology is 93.5%, specificity is 94.1%, and accuracy is 93.8%; for middle-aged physicians using TI-RADS classification combined with ultrasound S-Detect technology for diagnosis, the sensitivity is 89.6%, specificity is 92.2%, and accuracy is 90.6%; for junior doctors, the sensitivity of diagnosis using TI-RADS classification combined with ultrasound S-Detect technology is 83.1%, specificity is 88.2%, and accuracy is 85.1%. Regardless of seniority, the combined application of artificial intelligence ultrasound S-Detect technology and TI-RADS classification can improve the diagnostic ability of sonographers for thyroid nodules and at the same time improve the consistency of judgment among physicians, and this is especially important for radiologists.
Collapse
Affiliation(s)
- Peizhen Huang
- Department of Ultrasound and Imaging, Wenzhou Central Hospital, Wenzhou 325000, China
| | - Bin Zheng
- Wenzhou Medical University, Wenzhou 325000, China
| | - Mengyi Li
- Wenzhou Medical University, Wenzhou 325000, China
| | - Lin Xu
- Wenzhou Medical University, Wenzhou 325000, China
| | - Sajjad Rabbani
- Department of Electrical Engineering, Lahore College for Women University, LCWU, Lahore, Pakistan
| | | | | | - Beishu Zhan
- Department of Ultrasound and Imaging, Wenzhou Central Hospital, Wenzhou 325000, China
| | - He Jun
- Department of Ultrasound and Imaging, Wenzhou Central Hospital, Wenzhou 325000, China
| |
Collapse
|
13
|
Zhang X, Lee VCS, Rong J, Lee JC, Song J, Liu F. A multi-channel deep convolutional neural network for multi-classifying thyroid diseases. Comput Biol Med 2022; 148:105961. [PMID: 35985185 DOI: 10.1016/j.compbiomed.2022.105961] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2022] [Revised: 07/28/2022] [Accepted: 08/06/2022] [Indexed: 11/03/2022]
Abstract
BACKGROUND AND OBJECTIVE Thyroid disease instances have been continuously increasing since the 1990s, and thyroid cancer has become the most rapidly rising disease among all the malignancies in recent years. Most existing studies focused on applying deep convolutional neural networks for detecting thyroid cancer. Despite their satisfactory performance on binary classification tasks, limited studies have explored multi-class classification of thyroid disease types; much less is known of the diagnosis of co-existence situation for different types of thyroid diseases. METHOD This study proposed a novel multi-channel convolutional neural network (CNN) architecture to address the multi-class classification task of thyroid disease. The multi-channel CNN merits from computed tomography characteristics to drive a comprehensive diagnostic decision for the overall thyroid gland, emphasizing the disease co-existence circumstance. Moreover, this study also examined alternative strategies to enhance the diagnostic accuracy of CNN models through concatenation of different scales of feature maps. RESULTS Benchmarking experiments demonstrate the improved performance of the proposed multi-channel CNN architecture compared with the standard single-channel CNN architecture. More specifically, the multi-channel CNN achieved an accuracy of 0.909±0.048, precision of 0.944±0.062, recall of 0.896±0.047, specificity of 0.994±0.001, and F1 of 0.917±0.057, in contrast to the single-channel CNN, which obtained 0.902±0.004, 0.892±0.005, 0.909±0.002, 0.993±0.001, 0.898±0.003, respectively. In addition, the proposed model was evaluated in different gender groups; it reached a diagnostic accuracy of 0.908 for the female group and 0.901 for the male group. CONCLUSION Collectively, the results highlight that the proposed multi-channel CNN has excellent generalization and has the potential to be deployed to provide computational decision support in clinical settings.
Collapse
Affiliation(s)
- Xinyu Zhang
- Department of Data Science and AI, Faculty of IT, Monash University, Clayton, Melbourne, VIC 3800, Australia
| | - Vincent C S Lee
- Department of Data Science and AI, Faculty of IT, Monash University, Clayton, Melbourne, VIC 3800, Australia.
| | - Jia Rong
- Department of Data Science and AI, Faculty of IT, Monash University, Clayton, Melbourne, VIC 3800, Australia
| | - James C Lee
- Monash University Endocrine Surgery Unit, Alfred Hospital, Melbourne, VIC 3004, Australia; Department of Surgery, Monash University, Melbourne, VIC 3168, Australia
| | - Jiangning Song
- Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Clayton, Melbourne, VIC 3800, Australia
| | - Feng Liu
- West China Hospital of Sichuan University, Chengdu City, Sichuan Province 332001, China
| |
Collapse
|
14
|
Xie F, Luo YK, Lan Y, Tian XQ, Zhu YQ, Jin Z, Zhang Y, Zhang MB, Song Q, Zhang Y. Differential diagnosis and feature visualization for thyroid nodules using computer-aided ultrasonic diagnosis system: initial clinical assessment. BMC Med Imaging 2022; 22:153. [PMID: 36042395 PMCID: PMC9425995 DOI: 10.1186/s12880-022-00874-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Accepted: 08/16/2022] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND To assess the diagnostic efficacy of the computer-aided ultrasonic diagnosis system (CAD system) in differentiating benign and malignant thyroid nodules. METHODS The images of 296 thyroid nodules were included in validation sets. The diagnostic efficacy of the CAD system was compared with that of junior physicians and senior physicians, as well as that of the combination diagnosis of the CAD system with junior physicians. The diagnostic efficacy of the CAD system for different sizes of thyroid nodules was compared. RESULTS The diagnostic sensitivity and accuracy of the CAD system were higher than those of junior physicians (83.4% vs. 72.2%, 73.0% vs. 69.6%), but the diagnostic specificity of the CAD system was lower than that of junior physicians (62.1% vs. 66.9%). The diagnostic accuracy of the CAD system was lower than that of senior physicians (73.0% vs. 83.8%). However, the combination diagnosis of the CAD system with junior physicians had higher accuracy (81.8%) and AUC (0.842) than those of either the CAD system or junior physicians alone, and comparable diagnostic performance with those of senior physicians. The Kappa was 0.635 in the combination diagnosis of the CAD system with junior physicians, showing good consistency with the pathological results. The accuracy (76.4%) of the CAD system was the highest for nodules of 1-2 cm. CONCLUSION The CAD system can effectively assist physicians to identify malignant and benign thyroid nodules, reduce the overdiagnosis and overtreatment of thyroid nodules, avoid unnecessary invasive fine needle aspiration, and improve the diagnostic accuracy of junior physicians.
Collapse
Affiliation(s)
- Fang Xie
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Yu-Kun Luo
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Yu Lan
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Xiao-Qi Tian
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Ya-Qiong Zhu
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Zhuang Jin
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Ying Zhang
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Ming-Bo Zhang
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Qing Song
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| | - Yan Zhang
- grid.414252.40000 0004 1761 8894Department of Ultrasound, First Medical Center, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, 100853 China
| |
Collapse
|
15
|
Abstract
OBJECTIVES This meta-analysis aimed to evaluate the value of ultrasonic S-Detect mode for the evaluation of thyroid nodules. METHODS We searched PubMed, Cochrane Library, and Chinese biomedical databases from inception to August 31, 2021. Meta-analysis was conducted using STATA version 14.0 and Meta-Disc version 1.4 software. We calculated the summary statistics for sensitivity (Sen), specificity (Spe), summary receiver operating characteristic curve, and the area under the curve, and compared the area under the curve between ultrasonic S-Detect mode and thyroid imaging report and data system (TI-RADS) for the diagnosis of thyroid nodules. As a systematic review summarizing the results of previous studies, this study does not need the informed consent of patients or the approval of the ethics review committee. RESULTS Fifteen studies that met all inclusion criteria were included in this meta-analysis. A total of 924 thyroid malignant nodules and 1228 thyroid benign nodules were assessed. All thyroid nodules were histologically confirmed after examination. The pooled Sen and Spe of TI-RADS were 0.89 (95% confidence interval [CI] = 0.85-0.91) and 0.85 (95% CI = 0.78-0.90), respectively; the pooled Sen and Spe of S-Detect were 0.88 (95% CI = 0.85-0.90) and 0.73 (95% CI = 0.63-0.81), respectively. The areas under the summary receiver operating characteristic curve of TI-RADS and S-Detect were 0.9370 (standard error [SE] = 0.0110) and 0.9128 (SE = 0.0147), respectively, between which there was no significant difference (Z = 1.318; SE = 0.0184; P = .1875). We found no evidence of publication bias (t = 0.36, P = .72). CONCLUSIONS Our meta-analysis indicates that ultrasonic S-Detect mode may have high diagnostic accuracy and may have certain clinical application value, especially for young doctors.
Collapse
Affiliation(s)
- Jinyi Bian
- Ultrasound Department, The First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Ruyue Wang
- Dalian Medical University, Dalian, China
| | - Mingxin Lin
- Ultrasound Department, The First Affiliated Hospital of Dalian Medical University, Dalian, China
- *Correspondence: Mingxin Lin, Ultrasound Department, The First Affiliated Hospital of Dalian Medical University, No. 222 Zhongshan Road, Xigang District, Dalian City, Liaoning Province 116011, China (e-mail: )
| |
Collapse
|
16
|
Zhong L, Wang C. Diagnostic accuracy of S-Detect in distinguishing benign and malignant thyroid nodules: A meta-analysis. PLoS One 2022; 17:e0272149. [PMID: 35930525 PMCID: PMC9355179 DOI: 10.1371/journal.pone.0272149] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2021] [Accepted: 07/13/2022] [Indexed: 11/19/2022] Open
Abstract
Objectives In this meta-analysis study, the main objective was to determine the accuracy of S-detect in effectively distinguishing malignant thyroid nodules from benign thyroid nodules. Methods We searched the PubMed, Cochrane Library, and CBM databases from inception to August 1, 2021. Meta-analysis was conducted using STATA version 14.0 and Meta-Disc version 1.4 softwares. We calculated summary statistics for sensitivity (Sen), specificity (Spe), positive and negative likelihood ratio (LR+/LR−), diagnostic odds ratio(DOR), and receiver operating characteristic (SROC) curves. Cochran’s Q-statistic and I2 test were used to evaluate potential heterogeneity between studies. A sensitivity analysis was performed to evaluate the influence of single studies on the overall estimate. We also performed meta-regression analyses to investigate the potential sources of heterogeneity. Results In this study, a total of 17 studies meeting the requirements of the standard were used. The number of benign and malignant nodules analyzed and evaluated in this paper was 1595 and 1118 respectively. This paper mainly completes the required histological confirmation through s-detect. The pooled Sen and pooled Spe were 0.87 and 0.74, respectively, (95%CI = 0.84–0.89) and (95%CI = 0.66–0.81). Furthermore, the pooled LR+ and negative LR− were determined to be 3.37 (95%CI = 2.53–4.50) and 0.18 (95%CI = 0.15–0.21), respectively. The experimental results showed that the pooled DOR of thyroid nodules was 18.83 (95% CI = 13.21–26.84). In addition, area under SROC curve was determined to be 0.89 (SE = 0.0124). It should be pointed out that there is no evidence of bias (i.e. t = 0.25, P = 0.80). Conclusions Through this meta-analysis, it can be seen that the accuracy of s-detect is relatively high for the effective distinction between malignant thyroid nodules and benign thyroid nodules.
Collapse
Affiliation(s)
- Lin Zhong
- Pathology Department of the First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Cong Wang
- Ultrasound Department of the First Affiliated Hospital of Dalian Medical University, Dalian, China
- * E-mail:
| |
Collapse
|
17
|
Sorrenti S, Dolcetti V, Radzina M, Bellini MI, Frezza F, Munir K, Grani G, Durante C, D’Andrea V, David E, Calò PG, Lori E, Cantisani V. Artificial Intelligence for Thyroid Nodule Characterization: Where Are We Standing? Cancers (Basel) 2022; 14:cancers14143357. [PMID: 35884418 PMCID: PMC9315681 DOI: 10.3390/cancers14143357] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2022] [Revised: 06/24/2022] [Accepted: 07/08/2022] [Indexed: 12/12/2022] Open
Abstract
Simple Summary In the present review, an up-to-date summary of the state of the art of artificial intelligence (AI) implementation for thyroid nodule characterization and cancer is provided. The opinion on the real effectiveness of AI systems remains controversial. Taking into consideration the largest and most scientifically valid studies, it is possible to state that AI provides results that are comparable or inferior to expert ultrasound specialists and radiologists. Promising data approve AI as a support tool and simultaneously highlight the need for a radiologist supervisory framework for AI provided results. Therefore, current solutions might be more suitable for educational purposes. Abstract Machine learning (ML) is an interdisciplinary sector in the subset of artificial intelligence (AI) that creates systems to set up logical connections using algorithms, and thus offers predictions for complex data analysis. In the present review, an up-to-date summary of the current state of the art regarding ML and AI implementation for thyroid nodule ultrasound characterization and cancer is provided, highlighting controversies over AI application as well as possible benefits of ML, such as, for example, training purposes. There is evidence that AI increases diagnostic accuracy and significantly limits inter-observer variability by using standardized mathematical algorithms. It could also be of aid in practice settings with limited sub-specialty expertise, offering a second opinion by means of radiomics and computer-assisted diagnosis. The introduction of AI represents a revolutionary event in thyroid nodule evaluation, but key issues for further implementation include integration with radiologist expertise, impact on workflow and efficiency, and performance monitoring.
Collapse
Affiliation(s)
- Salvatore Sorrenti
- Department of Surgical Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (S.S.); (V.D.); (E.L.)
| | - Vincenzo Dolcetti
- Department of Radiological, Anatomo-Pathological Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (V.D.); (V.C.)
| | - Maija Radzina
- Radiology Research Laboratory, Riga Stradins University, LV-1007 Riga, Latvia;
- Medical Faculty, University of Latvia, Diagnostic Radiology Institute, Paula Stradina Clinical University Hospital, LV-1007 Riga, Latvia
| | - Maria Irene Bellini
- Department of Surgical Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (S.S.); (V.D.); (E.L.)
- Correspondence:
| | - Fabrizio Frezza
- Department of Information Engineering, Electronics and Telecommunications, “Sapienza” University of Rome, 00184 Rome, Italy; (F.F.); (K.M.)
- Consorzio Nazionale Interuniversitario per le Telecomunicazioni (CNIT), Viale G.P. Usberti 181/A Sede Scientifica di Ingegneria-Palazzina 3, 43124 Parma, Italy
| | - Khushboo Munir
- Department of Information Engineering, Electronics and Telecommunications, “Sapienza” University of Rome, 00184 Rome, Italy; (F.F.); (K.M.)
| | - Giorgio Grani
- Department of Translational and Precision Medicine, “Sapienza” University of Rome, 00161 Rome, Italy; (G.G.); (C.D.); (E.D.)
| | - Cosimo Durante
- Department of Translational and Precision Medicine, “Sapienza” University of Rome, 00161 Rome, Italy; (G.G.); (C.D.); (E.D.)
| | - Vito D’Andrea
- Department of Surgical Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (S.S.); (V.D.); (E.L.)
| | - Emanuele David
- Department of Translational and Precision Medicine, “Sapienza” University of Rome, 00161 Rome, Italy; (G.G.); (C.D.); (E.D.)
| | - Pietro Giorgio Calò
- Department of Surgical Sciences, “Policlinico Universitario Duilio Casula”, University of Cagliari, 09042 Monserrato, Italy;
| | - Eleonora Lori
- Department of Surgical Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (S.S.); (V.D.); (E.L.)
| | - Vito Cantisani
- Department of Radiological, Anatomo-Pathological Sciences, “Sapienza” University of Rome, 00161 Rome, Italy; (V.D.); (V.C.)
| |
Collapse
|
18
|
A comparison of artificial intelligence versus radiologists in the diagnosis of thyroid nodules using ultrasonography: a systematic review and meta-analysis. Eur Arch Otorhinolaryngol 2022; 279:5363-5373. [PMID: 35767056 DOI: 10.1007/s00405-022-07436-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 05/06/2022] [Indexed: 11/03/2022]
Abstract
BACKGROUND Thyroid nodules are common. Ultrasonography (US) is the first investigation for thyroid nodules. Artificial Intelligence (AI) is widely integrated into medical diagnosis to provide additional information. The primary objective of this study was to accumulate the pooled sensitivity and specificity between all available AI and radiologists using thyroid US imaging. The secondary objective was to compare AI's diagnostic performance to that of radiologists. MATERIALS AND METHODS A systematic review meta-analysis. PubMed, Scopus, Web of Science, and Cochrane Library data were searched for studies from inception until June 11, 2020. RESULTS Twenty five studies were included in this meta-analysis. The pooled sensitivity and specificity of AI were 0.86 (95% CI 0.81-0.91) and 0.78 (95% CI 0.73-0.83), respectively. The pooled sensitivity and specificity of radiologists were 0.85 (95% CI 0.80-0.89) and 0.82 (95% CI 0.77-0.86), respectively. The accuracy of AI and radiologists is equivalent in terms of AUC [AI 0.89 (95% CI 0.86-0.92), radiologist 0.91 (95% CI 0.88-0.93)]. The diagnostic odd ratio (DOR) between AI 23.10 (95% CI 14.20-37.58) and radiologists 27.12 (95% CI 17.45-42.16) had no statistically significant difference (P = 0.56). Meta-regression analysis revealed that Deep Learning AI had significantly greater sensitivity and specificity than classic machine learning AI (P < 0.001). CONCLUSION AI demonstrated comparable performance to radiologists in diagnosing benign and malignant thyroid nodules using ultrasonography. Additional research to establish its equivalency should be conducted.
Collapse
|
19
|
Yang K, Chen J, Wu H, Tian H, Ye X, Xu J, Luo X, Dong F. S-Thyroid Computer-Aided Diagnosis Ultrasound System of Thyroid Nodules: Correlation Between Transverse and Longitudinal Planes. Front Physiol 2022; 13:909277. [PMID: 35669572 PMCID: PMC9165693 DOI: 10.3389/fphys.2022.909277] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Accepted: 04/13/2022] [Indexed: 11/13/2022] Open
Abstract
Introduction: We compare the differences in the diagnostic results of S-thyroid, a computer-aided diagnosis (CAD) software, based on two mutually perpendicular planes. Methods: Initially, 149 thyroid nodules confirmed by surgical pathology were enrolled in our study. CAD in our study was based on the ACR TI-RADS lexicon. t test, rank-sum test, and Chi-square test were used. The interclass correlation coefficient and Cohen's kappa were used to explore the correlation between CAD features. Receiver operating characteristic was plotted for different combinations of CAD features. Results: The patient's age, transverse diameter, longitudinal diameter, shape, margin, echogenicity, echogenic foci, composition, TI-RADS classification, and risk probability of nodules in the transverse and longitudinal planes were related to thyroid cancer (p < 0.05). The AUC (95%CI) of TI-RADS classification in the transverse plane of CAD is better than that of the longitudinal plane [0.90 (0.84-0.95) vs. 0.83 (0.77-0.90), p = 0.04]. The AUC (95%CI) of risk probability of nodules in the transverse planes shows no difference from that in the longitudinal plane statistically [0.90 (0.85-0.95) vs. 0.88 (0.82-0.94), p = 0.52]. The AUC (95% CI), specificity, sensitivity, and accuracy [TI-RADS classification (transverse plane) + TI-RADS classification (longitudinal plane) + risk (transverse plane) + risk (longitudinal plane)] are 0.93 (0.89-0.97), 86.15%, 90.48%, and 88.59%, respectively. Conclusion: The diagnosis of thyroid cancer in the CAD transverse plane was superior to that in the CAD longitudinal plane when using the TI-RADS classification, but there was no difference in the diagnosis between the two planes when using risk. However, the combination of CAD transverse and longitudinal planes had the best diagnostic ability.
Collapse
Affiliation(s)
- Keen Yang
- The Second Clinical Medical College, Jinan University, Shenzhen, China
| | - Jing Chen
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Huaiyu Wu
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Hongtian Tian
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Xiuqin Ye
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Jinfeng Xu
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Xunpeng Luo
- Department of Thyroid Surgery, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| | - Fajin Dong
- Department of Ultrasound, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, China
| |
Collapse
|
20
|
Artificial Intelligence (AI) Tools for Thyroid Nodules on Ultrasound, From the AJR Special Series on AI Applications. AJR Am J Roentgenol 2022; 219:1-8. [PMID: 35383487 DOI: 10.2214/ajr.22.27430] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
Artificial intelligence (AI) methods for evaluating thyroid nodules on ultrasound have been widely described in the literature, with reported performance of AI tools matching or in some instances surpassing radiologists. As these data have accumulated, products for classification and risk stratification of thyroid nodules on ultrasound have become commercially available. This article reviews FDA-approved products currently on the market, with a focus on product features, reported performance, and considerations for implementation. The products perform risk stratification primarily using the Thyroid Imaging Reporting and Data System (TI-RADS), though may provide additional prediction tools independent of TI-RADS. Key issues in implementation include integration with radiologist interpretation, impact on workflow and efficiency, and performance monitoring. AI applications beyond nodule classification, including report construction and incidental findings follow-up, are also described. Anticipated future directions of research and development in AI tools for thyroid nodules are highlighted.
Collapse
|
21
|
Cleere EF, Davey MG, O’Neill S, Corbett M, O’Donnell JP, Hacking S, Keogh IJ, Lowery AJ, Kerin MJ. Radiomic Detection of Malignancy within Thyroid Nodules Using Ultrasonography-A Systematic Review and Meta-Analysis. Diagnostics (Basel) 2022; 12:diagnostics12040794. [PMID: 35453841 PMCID: PMC9027085 DOI: 10.3390/diagnostics12040794] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 03/22/2022] [Accepted: 03/22/2022] [Indexed: 02/04/2023] Open
Abstract
Background: Despite investigation, 95% of thyroid nodules are ultimately benign. Radiomics is a field that uses radiological features to inform individualized patient care. We aimed to evaluate the diagnostic utility of radiomics in classifying undetermined thyroid nodules into benign and malignant using ultrasonography (US). Methods: A diagnostic test accuracy systematic review and meta-analysis was performed in accordance with PRISMA guidelines. Sensitivity, specificity, and area under curve (AUC) delineating benign and malignant lesions were recorded. Results: Seventy-five studies including 26,373 patients and 46,175 thyroid nodules met inclusion criteria. Males accounted for 24.6% of patients, while 75.4% of patients were female. Radiomics provided a pooled sensitivity of 0.87 (95% CI: 0.86−0.87) and a pooled specificity of 0.84 (95% CI: 0.84−0.85) for characterizing benign and malignant lesions. Using convolutional neural network (CNN) methods, pooled sensitivity was 0.85 (95% CI: 0.84−0.86) and pooled specificity was 0.82 (95% CI: 0.82−0.83); significantly lower than studies using non-CNN: sensitivity 0.90 (95% CI: 0.89−0.90) and specificity 0.88 (95% CI: 0.87−0.89) (p < 0.05). The diagnostic ability of radiologists and radiomics were comparable for both sensitivity (OR 0.98) and specificity (OR 0.95). Conclusions: Radiomic analysis using US provides a reproducible, reliable evaluation of undetermined thyroid nodules when compared to current best practice.
Collapse
Affiliation(s)
- Eoin F. Cleere
- The Lambe Institute for Translational Research, National University of Ireland, H91 YR71 Galway, Ireland; (M.G.D.); (A.J.L.); (M.J.K.)
- Department of Otolaryngology, Galway University Hospitals, H91 YR71 Galway, Ireland; (M.C.); (I.J.K.)
- Correspondence:
| | - Matthew G. Davey
- The Lambe Institute for Translational Research, National University of Ireland, H91 YR71 Galway, Ireland; (M.G.D.); (A.J.L.); (M.J.K.)
| | - Shane O’Neill
- Department of Breast and Endocrine Surgery, Galway University Hospitals, H91 YR71 Galway, Ireland;
| | - Mel Corbett
- Department of Otolaryngology, Galway University Hospitals, H91 YR71 Galway, Ireland; (M.C.); (I.J.K.)
| | - John P O’Donnell
- Department of Radiology, Galway University Hospitals, H91 YR71 Galway, Ireland;
| | - Sean Hacking
- Department of Pathology and Laboratory Medicine, Warren Alpert Medical School of Brown University, Providence, RI 02903, USA;
| | - Ivan J. Keogh
- Department of Otolaryngology, Galway University Hospitals, H91 YR71 Galway, Ireland; (M.C.); (I.J.K.)
| | - Aoife J. Lowery
- The Lambe Institute for Translational Research, National University of Ireland, H91 YR71 Galway, Ireland; (M.G.D.); (A.J.L.); (M.J.K.)
- Department of Breast and Endocrine Surgery, Galway University Hospitals, H91 YR71 Galway, Ireland;
| | - Michael J. Kerin
- The Lambe Institute for Translational Research, National University of Ireland, H91 YR71 Galway, Ireland; (M.G.D.); (A.J.L.); (M.J.K.)
- Department of Breast and Endocrine Surgery, Galway University Hospitals, H91 YR71 Galway, Ireland;
| |
Collapse
|
22
|
Chambara N, Liu SYW, Lo X, Ying M. Comparative Analysis of Computer-Aided Diagnosis and Computer-Assisted Subjective Assessment in Thyroid Ultrasound. Life (Basel) 2021; 11:life11111148. [PMID: 34833024 PMCID: PMC8621517 DOI: 10.3390/life11111148] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Revised: 10/21/2021] [Accepted: 10/25/2021] [Indexed: 12/17/2022] Open
Abstract
The value of computer-aided diagnosis (CAD) and computer-assisted techniques equipped with different TIRADS remains ambiguous. Parallel diagnosis performances of computer-assisted subjective assessments and CAD were compared based on AACE, ATA, EU, and KSThR TIRADS. CAD software computed the diagnosis of 162 thyroid nodule sonograms. Two raters (R1 and R2) independently rated the sonographic features of the nodules using an online risk calculator while blinded to pathology results. Diagnostic efficiency measures were calculated based on the final pathology results. R1 had higher diagnostic performance outcomes than CAD with similarities between KSThR (SEN: 90.3% vs. 83.9%, p = 0.57; SPEC: 46% vs. 51%, p = 0.21; AUROC: 0.76 vs. 0.67, p = 0.02), and EU (SEN: 85.5% vs. 79%, p = 0.82; SPEC: 62% vs. 55%, p = 0.27; AUROC: 0.74 vs. 0.67, p = 0.06). Similarly, R2 had higher AUROC and specificity but lower sensitivity than CAD (KSThR-AUROC: 0.74 vs. 0.67, p = 0.13; SPEC: 61% vs. 46%, p = 0.02 and SEN: 75.8% vs. 83.9%, p = 0.31, and EU-AUROC: 0.69 vs. 0.67, p = 0.57, SPEC: 64% vs. 55%, p = 0.19, and SEN: 71% vs. 79%, p = 0.51, respectively). CAD had higher sensitivity but lower specificity than both R1 and R2 with AACE for 114 specified nodules (SEN: 92.5% vs. 88.7%, p = 0.50; 92.5% vs. 79.3%, p = 0.02, and SPEC: 26.2% vs. 54.1%, p = 0.001; 26.2% vs. 62.3%, p < 0.001, respectively). All diagnostic performance outcomes were comparable for ATA with 96 specified nodules. Computer-assisted subjective interpretation using KSThR is more ideal for ruling out papillary thyroid carcinomas than CAD. Future larger multi-center and multi-rater prospective studies with a diverse representation of thyroid cancers are necessary to validate these findings.
Collapse
Affiliation(s)
- Nonhlanhla Chambara
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;
| | - Shirley Yuk Wah Liu
- Department of Surgery, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, New Territories, Hong Kong, China;
| | - Xina Lo
- Department of Surgery, North District Hospital, Sheung Shui, New Territories, Hong Kong, China;
| | - Michael Ying
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;
- Correspondence: ; Tel.: +852-3400-8566
| |
Collapse
|
23
|
Cordes M, Götz TI, Lang EW, Coerper S, Kuwert T, Schmidkonz C. Advanced thyroid carcinomas: neural network analysis of ultrasonographic characteristics. Thyroid Res 2021; 14:16. [PMID: 34187534 PMCID: PMC8240264 DOI: 10.1186/s13044-021-00107-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/03/2021] [Accepted: 06/21/2021] [Indexed: 12/20/2022] Open
Abstract
BACKGROUND Ultrasound is the first-line imaging modality for detection and classification of thyroid nodules. Certain characteristics observable by ultrasound have recently been identified that may indicate malignancy. This retrospective cohort study was conducted to test the hypothesis that advanced thyroid carcinomas show distinctive clinical and sonographic characteristics. Using a neural network model as proof of concept, nine clinical/sonographic features served as input. METHODS All 96 study enrollees had histologically confirmed thyroid carcinomas, categorized (n = 32, each) as follows: group 1, advanced carcinoma (ADV) marked by local invasion or distant metastasis; group 2, non-advanced papillary carcinoma (PTC); or group 3, non-advanced follicular carcinoma (FTC). Preoperative ultrasound profiles were obtained via standardized protocols. The neural network had nine input neurons and one hidden layer. RESULTS Mean age and the number of male patients in group 1 were significantly higher compared with groups 2 (p = 0.005) or 3 (p < 0.001). On ultrasound, tumors of larger volume and irregular shape were observed significantly more often in group 1 compared with groups 2 (p < 0.001) or 3 (p ≤ 0.01). Network accuracy in discriminating advanced vs. non-advanced tumors was 84.4% (95% confidence interval [CI]: 75.5-91), with positive and negative predictive values of 87.1% (95% CI: 70.2-96.4) and 92.3% (95% CI: 83.0-97.5), respectively. CONCLUSIONS Our study has shown some evidence that advanced thyroid tumors demonstrate distinctive clinical and sonographic characteristics. Further prospective investigations with larger numbers of patients and multicenter design should be carried out to show whether a neural network incorporating these features may be an asset, helping to classify malignancies of the thyroid gland.
Collapse
Affiliation(s)
- Michael Cordes
- Radiologisch-Nuklearmedizinisches Zentrum, Martin-Richter-Str. 43, 90489, Nürnberg, Germany. .,Nuklearmedizinische Klinik, Universitätsklinikum Erlangen, Erlangen, Germany.
| | - Theresa Ida Götz
- CIML Group, Biophysics, University of Regensburg, Regensburg, Germany
| | | | - Stephan Coerper
- Klinik für Allgemein- und Viszeralchirurgie, Krankenhaus Martha-Maria, Nürnberg, Germany
| | - Torsten Kuwert
- Nuklearmedizinische Klinik, Universitätsklinikum Erlangen, Erlangen, Germany
| | | |
Collapse
|
24
|
Kim GR, Lee E, Kim HR, Yoon JH, Park VY, Kwak JY. Convolutional Neural Network to Stratify the Malignancy Risk of Thyroid Nodules: Diagnostic Performance Compared with the American College of Radiology Thyroid Imaging Reporting and Data System Implemented by Experienced Radiologists. AJNR Am J Neuroradiol 2021; 42:1513-1519. [PMID: 33985947 DOI: 10.3174/ajnr.a7149] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2020] [Accepted: 03/06/2021] [Indexed: 12/12/2022]
Abstract
BACKGROUND AND PURPOSE Comparison of the diagnostic performance for thyroid cancer on ultrasound between a convolutional neural network and visual assessment by radiologists has been inconsistent. Thus, we aimed to evaluate the diagnostic performance of the convolutional neural network compared with the American College of Radiology Thyroid Imaging Reporting and Data System (TI-RADS) for the diagnosis of thyroid cancer using ultrasound images. MATERIALS AND METHODS From March 2019 to September 2019, seven hundred sixty thyroid nodules (≥10 mm) in 757 patients were diagnosed as benign or malignant through fine-needle aspiration, core needle biopsy, or an operation. Experienced radiologists assessed the sonographic descriptors of the nodules, and 1 of 5 American College of Radiology TI-RADS categories was assigned. The convolutional neural network provided malignancy risk percentages for nodules based on sonographic images. Sensitivity, specificity, accuracy, positive predictive value, and negative predictive value were calculated with cutoff values using the Youden index and compared between the convolutional neural network and the American College of Radiology TI-RADS. Areas under the receiver operating characteristic curve were also compared. RESULTS Of 760 nodules, 176 (23.2%) were malignant. At an optimal threshold derived from the Youden index, sensitivity and negative predictive values were higher with the convolutional neural network than with the American College of Radiology TI-RADS (81.8% versus 73.9%, P = .009; 94.0% versus 92.2%, P = .046). Specificity, accuracy, and positive predictive values were lower with the convolutional neural network than with the American College of Radiology TI-RADS (86.1% versus 93.7%, P < .001; 85.1% versus 89.1%, P = .003; and 64.0% versus 77.8%, P < .001). The area under the curve of the convolutional neural network was higher than that of the American College of Radiology TI-RADS (0.917 versus 0.891, P = .017). CONCLUSIONS The convolutional neural network provided diagnostic performance comparable with that of the American College of Radiology TI-RADS categories assigned by experienced radiologists.
Collapse
Affiliation(s)
- G R Kim
- From the Department of Radiology (G.R.K., J.H.Y., V.Y.P., J.Y.K.), Severance Hospital, Research Institute of Radiological Science, Center for Clinical Imaging Data Science, Yonsei University College of Medicine, Seoul, Korea
| | - E Lee
- Department of Computational Science and Engineering (E.L.), Yonsei University, Seoul, Korea
| | - H R Kim
- Biostatistics Collaboration Unit (H.R.K.), Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Korea
| | - J H Yoon
- From the Department of Radiology (G.R.K., J.H.Y., V.Y.P., J.Y.K.), Severance Hospital, Research Institute of Radiological Science, Center for Clinical Imaging Data Science, Yonsei University College of Medicine, Seoul, Korea
| | - V Y Park
- From the Department of Radiology (G.R.K., J.H.Y., V.Y.P., J.Y.K.), Severance Hospital, Research Institute of Radiological Science, Center for Clinical Imaging Data Science, Yonsei University College of Medicine, Seoul, Korea
| | - J Y Kwak
- From the Department of Radiology (G.R.K., J.H.Y., V.Y.P., J.Y.K.), Severance Hospital, Research Institute of Radiological Science, Center for Clinical Imaging Data Science, Yonsei University College of Medicine, Seoul, Korea
| |
Collapse
|
25
|
Shao Y, Zhang YX, Chen HH, Lu SS, Zhang SC, Zhang JX. Advances in the application of artificial intelligence in solid tumor imaging. Artif Intell Cancer 2021; 2:12-24. [DOI: 10.35713/aic.v2.i2.12] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Revised: 04/02/2021] [Accepted: 04/20/2021] [Indexed: 02/06/2023] Open
Abstract
Early diagnosis and timely treatment are crucial in reducing cancer-related mortality. Artificial intelligence (AI) has greatly relieved clinical workloads and changed the current medical workflows. We searched for recent studies, reports and reviews referring to AI and solid tumors; many reviews have summarized AI applications in the diagnosis and treatment of a single tumor type. We herein systematically review the advances of AI application in multiple solid tumors including esophagus, stomach, intestine, breast, thyroid, prostate, lung, liver, cervix, pancreas and kidney with a specific focus on the continual improvement on model performance in imaging practice.
Collapse
Affiliation(s)
- Ying Shao
- Department of Laboratory Medicine, People Hospital of Jiangying, Jiangying 214400, Jiangsu Province, China
| | - Yu-Xuan Zhang
- Department of Laboratory Medicine, The First Affiliated Hospital of Nanjing Medical University, Nanjing 210029, Jiangsu Province, China
| | - Huan-Huan Chen
- Department of Laboratory Medicine, The First Affiliated Hospital of Nanjing Medical University, Nanjing 210029, Jiangsu Province, China
| | - Shan-Shan Lu
- Department of Radiology, The First Affiliated Hospital of Nanjing Medical University, Nanjing 210029, Jiangsu Province, China
| | - Shi-Chang Zhang
- Department of Laboratory Medicine, The First Affiliated Hospital of Nanjing Medical University, Nanjing 210029, Jiangsu Province, China
| | - Jie-Xin Zhang
- Department of Laboratory Medicine, The First Affiliated Hospital of Nanjing Medical University, Nanjing 210029, Jiangsu Province, China
| |
Collapse
|
26
|
Jin Z, Zhu Y, Zhang S, Xie F, Zhang M, Guo Y, Wang H, Zhu Q, Cao J, Luo Y. Diagnosis of thyroid cancer using a TI-RADS-based computer-aided diagnosis system: a multicenter retrospective study. Clin Imaging 2021; 80:43-49. [PMID: 34237590 DOI: 10.1016/j.clinimag.2020.12.012] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2020] [Revised: 11/18/2020] [Accepted: 12/01/2020] [Indexed: 10/22/2022]
Abstract
OBJECTIVE The purpose of this study was to use a computer-aided diagnosis (CAD) system based on the Thyroid Imaging, Reporting, and Data System (TI-RADS) to improve the diagnostic performance of thyroid cancer by analyzing clinical ultrasound imaging data. METHODS A retrospective diagnostic study of ultrasound image sets was conducted at five hospitals in China. A CAD system based on TI-RADS was applied in this study, and the diagnostic performance of CAD system was tested through multi-center data. The performance of the CAD system was compared with the consensus of three experienced radiologists. The interobserver agreement for cancer diagnosis was calculated between the CAD system and the consensus of the three experienced radiologists. RESULTS The CAD system performed well in the diagnosis of thyroid cancer, with an area under the curve (AUC) value of 0.902 (95% CI: 0.884-0.918), and obtained results similar to those of the three experienced radiologists. The CAD system performed better in the internal test set than in the external test set (AUC: 0.930 vs 0.877, respectively). The performance of the CAD system in the diagnosis of thyroid cancer for nodules of different sizes (<1 cm, 1-2 cm and ≥2 cm) was basically similar (accuracy: 84.6% vs 85% vs 84.2%). The CAD system can recognize 15 ultrasound features of thyroid nodules, most of which reached the level of 3 experienced radiologists (12/15, 85%). CONCLUSION The CAD system achieved an improved AUC and similar sensitivity and specificity in the diagnosis of thyroid cancer compared with the consensus of experienced radiologists.
Collapse
Affiliation(s)
- Zhuang Jin
- Department of Ultrasound, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, China
| | - Yaqiong Zhu
- Department of Ultrasound, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, China; Nankai University, No. 94 Weijin Road, Nankai District, Tianjin City, China
| | - Shijie Zhang
- Peking University, No. 5 Yiheyuan Road, Haidian District, Beijing 10087, China
| | - Fang Xie
- Department of Ultrasound, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, China
| | - Mingbo Zhang
- Department of Ultrasound, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, China
| | - Yanli Guo
- Department of Ultrasound, Southwest Hospital, Third Military Medical University (Army Medical University), Shapingba District, Chongqing, China
| | - Hui Wang
- Department of Ultrasound, China-Japan Union Hospital of Jilin University, Changchun 130000, Jilin, China
| | - Qiang Zhu
- Department of Diagnostic Ultrasound, Beijing Tongren Hospital, Capital Medical University, Beijing, China
| | - Junying Cao
- Department of Ultrasound, General Hospital of Northern Theater Command, No. 83, Wenhua Road, Shenhe District, Shenyang, Liaoning Province 110018, China.
| | - Yukun Luo
- Department of Ultrasound, Chinese PLA General Hospital, No. 28, Fuxing Road, Haidian District, Beijing, China.
| |
Collapse
|
27
|
Li LR, Du B, Liu HQ, Chen C. Artificial Intelligence for Personalized Medicine in Thyroid Cancer: Current Status and Future Perspectives. Front Oncol 2021; 10:604051. [PMID: 33634025 PMCID: PMC7899964 DOI: 10.3389/fonc.2020.604051] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2020] [Accepted: 12/21/2020] [Indexed: 12/12/2022] Open
Abstract
Thyroid cancers (TC) have increasingly been detected following advances in diagnostic methods. Risk stratification guided by refined information becomes a crucial step toward the goal of personalized medicine. The diagnosis of TC mainly relies on imaging analysis, but visual examination may not reveal much information and not enable comprehensive analysis. Artificial intelligence (AI) is a technology used to extract and quantify key image information by simulating complex human functions. This latent, precise information contributes to stratify TC on the distinct risk and drives tailored management to transit from the surface (population-based) to a point (individual-based). In this review, we started with several challenges regarding personalized care in TC, for example, inconsistent rating ability of ultrasound physicians, uncertainty in cytopathological diagnosis, difficulty in discriminating follicular neoplasms, and inaccurate prognostication. We then analyzed and summarized the advances of AI to extract and analyze morphological, textural, and molecular features to reveal the ground truth of TC. Consequently, their combination with AI technology will make individual medical strategies possible.
Collapse
Affiliation(s)
- Ling-Rui Li
- Department of Breast and Thyroid Surgery, Renmin Hospital of Wuhan University, Wuhan, China
| | - Bo Du
- School of Computer Science, Wuhan University, Wuhan, China.,Institute of Artificial Intelligence, Wuhan University, Wuhan, China
| | - Han-Qing Liu
- Department of Breast and Thyroid Surgery, Renmin Hospital of Wuhan University, Wuhan, China
| | - Chuang Chen
- Department of Breast and Thyroid Surgery, Renmin Hospital of Wuhan University, Wuhan, China
| |
Collapse
|
28
|
Lian KM, Lin T. Value of image-pro plus for assisting virtual touch tissue imaging in the diagnosis of thyroid nodules. Clin Hemorheol Microcirc 2021; 77:143-151. [PMID: 33185591 DOI: 10.3233/ch-200983] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
PURPOSE The value of virtual touch tissue imaging (VTI) with support of Image-Pro Plus (IPP) for diagnosing malignant thyroid tumors was assessed in the present study. METHODS In this retrospective study, we enrolled 160 patients with 198 thyroid nodules. TI-RADS, VTI grade, and VTI with support of IPP (VTI-IPP) were underwent for each nodule. With the pathological diagnosis as the gold standard, the receiver-operating characteristic curve (ROC) was drawn to evaluate the diagnostic performance of VTI-IPP, VTI, TI-RADS, VTI-IPP combinate with TI-RADS in thyroid carcinoma. RESULTS VTI-IPP score >2, VTI score >3, TI-RADS score >1, and VTI-IPP combine with TI-RADS score >4 expressed the highest diagnostic value for malignant thyroid nodules, the areas under the curve (AUC) were 0.939, 0.905, 0.925, and 0.967, respectively. The combination indicated the largest AUC, compared with VTI-IPP and TI-RADS, respectively (P = 0.0054 and 0.0009). The performance of VTI-IPP in diagnosing thyroid carcinomas was better than VTI (P = 0.0321). CONCLUSION Compare with VTI, VTI-IPP exhibited more excellent value in distinguishing between benign and malignant thyroid nodules. The value of malignant thyroid nodules diagnosis can be improved when VTI-IPP combines with TI-RADS.
Collapse
Affiliation(s)
- Kai-Mei Lian
- Department of Ultrasound, The First Affiliated Hospital of Shantou University Medical College, Shantou, Guangdong Province, P.R. China
| | - Teng Lin
- Department of Ultrasound, The First Affiliated Hospital of Shantou University Medical College, Shantou, Guangdong Province, P.R. China
| |
Collapse
|
29
|
Ye FY, Lyu GR, Li SQ, You JH, Wang KJ, Cai ML, Su QC. Diagnostic Performance of Ultrasound Computer-Aided Diagnosis Software Compared with That of Radiologists with Different Levels of Expertise for Thyroid Malignancy: A Multicenter Prospective Study. ULTRASOUND IN MEDICINE & BIOLOGY 2021; 47:114-124. [PMID: 33239154 DOI: 10.1016/j.ultrasmedbio.2020.09.019] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2020] [Revised: 09/19/2020] [Accepted: 09/22/2020] [Indexed: 06/11/2023]
Abstract
The aim of the work described here was to evaluate the diagnostic performance of ultrasound thyroid computer-aided diagnosis (CAD) software. This multicenter prospective study included 494 patients (565 thyroid nodules) who underwent surgery or biopsy after ultrasonography at four hospitals from January 2019 to September 2019. The diagnostic performance metrics of different readers were calculated and compared with the pathologic results. The sensitivity of CAD was outstanding and was equivalent to that of a senior radiologist (90.51% vs. 88.47%, p > 0.05). The area under the curve of CAD was equivalent to that of a junior radiologist (0.748 vs. 0.739, p > 0.05). However, the specificity was only 49.63%, which was lower than those of the three radiologists (75.56%, 85.93% and 90.37% for the junior, intermediate and senior radiologists, respectively). The diagnostic performance of the junior radiologist was significantly improved with the aid of CAD (junior + CAD). The sensitivity and area under the curve of junior + CAD were improved from 72.20% to 89.93% and from 0.739 to 0.816, respectively (both p values <0.05), and the positive predictive value, negative predictive value and κ coefficient improved from 76.3% to 78.6%, 82.0% to 86.8% and 0.394 to 0.511, respectively. Though specificity slightly decreased from 75.56% to 73.33%, the difference was not statistically significant (p > 0.05). In general, the clinical application value of CAD is promising, and its instrumental value for junior radiologists is significant.
Collapse
Affiliation(s)
- Feng-Ying Ye
- Department of Ultrasound, Second Affiliated Hospital of Fujian Medical University, Quanzhou, China
| | - Guo-Rong Lyu
- Department of Clinical Medicine, Quanzhou Medical College, Quanzhou, China.
| | - Shang-Qing Li
- Department of Clinical Medicine, Quanzhou Medical College, Quanzhou, China
| | - Jian-Hong You
- Department of Ultrasound, Zhongshan Hospital Affiliated to Xiamen University, Xiamen, China
| | - Kang-Jian Wang
- Department of Ultrasound, Zhangzhou Affiliated Hospital of Fujian Medical University, Zhangzhou, China
| | - Ming-Li Cai
- Department of Ultrasound, Jinjiang City Hospital, Jinjiang, China
| | - Qi-Chen Su
- Department of Ultrasound, Second Affiliated Hospital of Fujian Medical University, Quanzhou, China
| |
Collapse
|
30
|
Ha EJ, Baek JH. Applications of machine learning and deep learning to thyroid imaging: where do we stand? Ultrasonography 2021; 40:23-29. [PMID: 32660203 PMCID: PMC7758100 DOI: 10.14366/usg.20068] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Revised: 07/01/2020] [Accepted: 07/03/2020] [Indexed: 01/17/2023] Open
Abstract
Ultrasonography (US) is the primary diagnostic tool used to assess the risk of malignancy and to inform decision-making regarding the use of fine-needle aspiration (FNA) and postFNA management in patients with thyroid nodules. However, since US image interpretation is operator-dependent and interobserver variability is moderate to substantial, unnecessary FNA and/or diagnostic surgery are common in practice. Artificial intelligence (AI)-based computeraided diagnosis (CAD) systems have been introduced to help with the accurate and consistent interpretation of US features, ultimately leading to a decrease in unnecessary FNA. This review provides a developmental overview of the AI-based CAD systems currently used for thyroid nodules and describes the future developmental directions of these systems for the personalized and optimized management of thyroid nodules.
Collapse
Affiliation(s)
- Eun Ju Ha
- Department of Radiology, Ajou University School of Medicine, Suwon, Korea
| | - Jung Hwan Baek
- Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
| |
Collapse
|
31
|
Han M, Ha EJ, Park JH. Computer-Aided Diagnostic System for Thyroid Nodules on Ultrasonography: Diagnostic Performance Based on the Thyroid Imaging Reporting and Data System Classification and Dichotomous Outcomes. AJNR Am J Neuroradiol 2020; 42:559-565. [PMID: 33361374 DOI: 10.3174/ajnr.a6922] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Accepted: 09/29/2020] [Indexed: 01/19/2023]
Abstract
BACKGROUND AND PURPOSE Artificial intelligence-based computer-aided diagnostic systems have been introduced for thyroid cancer diagnosis. Our aim was to compare the diagnostic performance of a commercially available computer-aided diagnostic system and radiologist-based assessment for the detection of thyroid cancer based on the Thyroid Imaging Reporting and Data Systems (TIRADS) and dichotomous outcomes. MATERIALS AND METHODS In total, 372 consecutive patients with 454 thyroid nodules were enrolled. The computer-aided diagnostic system was set up to render a possible diagnosis in 2 formats, the Korean Society of Thyroid Radiology (K)-TIRADS and the American Thyroid Association (ATA)-TIRADS-classifications, and dichotomous outcomes (possibly benign or possibly malignant). RESULTS The diagnostic sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the computer-aided diagnostic system for thyroid cancer were, respectively, 97.6%, 21.6%, 42.0%, 93.9%, and 49.6% for K-TIRADS; 94.6%, 29.6%, 43.9%, 90.4%, and 53.5% for ATA-TIRADS; and 81.4%, 81.9%, 72.3%, 88.3%, and 81.7% for dichotomous outcomes. The sensitivities of the computer-aided diagnostic system did not differ significantly from those of the radiologist (all P > .05); the specificities and accuracies were significantly lower than those of the radiologist (all P < .001). Unnecessary fine-needle aspiration rates were lower for the dichotomous outcome characterizations, particularly for those performed by the radiologist. The interobserver agreement for the description of K-TIRADS and ATA-TIRADS classifications was fair-to-moderate, but the dichotomous outcomes were in substantial agreement. CONCLUSIONS The diagnostic performance of the computer-aided diagnostic system varies in terms of TIRADS classification and dichotomous outcomes and relative to radiologist-based assessments. Clinicians should know about the strengths and weaknesses associated with the diagnosis of thyroid cancer using computer-aided diagnostic systems.
Collapse
Affiliation(s)
- M Han
- Department of Radiology, Ajou University School of Medicine, Suwon, Korea
| | - E J Ha
- Department of Radiology, Ajou University School of Medicine, Suwon, Korea
| | - J H Park
- Department of Radiology, Ajou University School of Medicine, Suwon, Korea
| |
Collapse
|
32
|
Chung SR, Baek JH, Lee MK, Ahn Y, Choi YJ, Sung TY, Song DE, Kim TY, Lee JH. Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists. Korean J Radiol 2020; 21:369-376. [PMID: 32090529 PMCID: PMC7039724 DOI: 10.3348/kjr.2019.0581] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2019] [Accepted: 11/04/2019] [Indexed: 12/27/2022] Open
Abstract
Objective To determine whether a computer-aided diagnosis (CAD) system for the evaluation of thyroid nodules is non-inferior to radiologists with different levels of experience. Materials and Methods Patients with thyroid nodules with a decisive diagnosis of benign or malignant nodule were consecutively enrolled from November 2017 to September 2018. Three radiologists with different levels of experience (1 month, 4 years, and 7 years) in thyroid ultrasound (US) reviewed the thyroid US with and without using the CAD system. Statistical analyses included non-inferiority testing of the diagnostic accuracy for malignant thyroid nodules between the CAD system and the three radiologists with a non-inferiority margin of 10%, comparison of the diagnostic performance, and the added value of the CAD system to the radiologists. Results Altogether, 197 patients were included in the study cohort. The diagnostic accuracy of the CAD system (88.48%, 95% confidence interval [CI] = 82.65–92.53) was non-inferior to that of the radiologists with less experience (1 month and 4 year) of thyroid US (83.03%, 95% CI = 76.52–88.02; p < 0.001), whereas it was inferior to that of the experienced radiologist (7 years) (95.76%, 95% CI = 91.37–97.96; p = 0.138). The sensitivity and negative predictive value of the CAD system were significantly higher than those of the less-experienced radiologists were, whereas no significant difference was found with those of the experienced radiologist. A combination of US and the CAD system significantly improved sensitivity and negative predictive value, although the specificity and positive predictive value deteriorated for the less-experienced radiologists. Conclusion The CAD system may offer support for decision-making in the diagnosis of malignant thyroid nodules for operators who have less experience with thyroid US.
Collapse
Affiliation(s)
- Sae Rom Chung
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Jung Hwan Baek
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.
| | - Min Kyoung Lee
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Yura Ahn
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Young Jun Choi
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Tae Yon Sung
- Department of Surgery, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Dong Eun Song
- Department of Pathology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Tae Yong Kim
- Department of Endocrinology and Metabolism, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Jeong Hyun Lee
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| |
Collapse
|
33
|
S-Detect Software vs. EU-TIRADS Classification: A Dual-Center Validation of Diagnostic Performance in Differentiation of Thyroid Nodules. J Clin Med 2020; 9:jcm9082495. [PMID: 32756510 PMCID: PMC7464710 DOI: 10.3390/jcm9082495] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Revised: 07/23/2020] [Accepted: 07/30/2020] [Indexed: 12/12/2022] Open
Abstract
Computer-aided diagnosis (CAD) and other risk stratification systems may improve ultrasound image interpretation. This prospective study aimed to compare the diagnostic performance of CAD and the European Thyroid Imaging Reporting and Data System (EU-TIRADS) classification applied by physicians with S-Detect 2 software CAD based on Korean Thyroid Imaging Reporting and Data System (K-TIRADS) and combinations of both methods (MODELs 1 to 5). In all, 133 nodules from 88 patients referred to thyroidectomy with available histopathology or with unambiguous results of cytology were included. The S-Detect system, EU-TIRADS, and mixed MODELs 1–5 for the diagnosis of thyroid cancer showed a sensitivity of 89.4%, 90.9%, 84.9%, 95.5%, 93.9%, 78.9% and 93.9%; a specificity of 80.6%, 61.2%, 88.1%, 53.7%, 73.1%, 89.6% and 80.6%; a positive predictive value of 81.9%, 69.8%, 87.5%, 67%, 77.5%, 88.1% and 82.7%; a negative predictive value of 88.5%, 87.2%, 85.5%, 92.3%, 92.5%, 81.1% and 93.1%; and an accuracy of 85%, 75.9%, 86.5%, 74.4%, 83.5%, 84.2%, and 87.2%, respectively. Comparison showed superiority of the similar MODELs 1 and 5 over other mixed models as well as EU-TIRADS and S-Detect used alone (p-value < 0.05). S-Detect software is characterized with high sensitivity and good specificity, whereas EU-TIRADS has high sensitivity, but rather low specificity. The best diagnostic performance in malignant thyroid nodule (TN) risk stratification was obtained for the combined model of S-Detect (“possibly malignant” nodule) and simultaneously obtaining 4 or 5 points (MODEL 1) or exactly 5 points (MODEL 5) on the EU-TIRADS scale.
Collapse
|
34
|
Xu L, Gao J, Wang Q, Yin J, Yu P, Bai B, Pei R, Chen D, Yang G, Wang S, Wan M. Computer-Aided Diagnosis Systems in Diagnosing Malignant Thyroid Nodules on Ultrasonography: A Systematic Review and Meta-Analysis. Eur Thyroid J 2020; 9:186-193. [PMID: 32903956 PMCID: PMC7445671 DOI: 10.1159/000504390] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2019] [Revised: 10/25/2019] [Indexed: 12/16/2022] Open
Abstract
BACKGROUND Computer-aided diagnosis (CAD) systems are being applied to the ultrasonographic diagnosis of malignant thyroid nodules, but it remains controversial whether the systems add any accuracy for radiologists. OBJECTIVE To determine the accuracy of CAD systems in diagnosing malignant thyroid nodules. METHODS PubMed, EMBASE, and the Cochrane Library were searched for studies on the diagnostic performance of CAD systems. The diagnostic performance was assessed by pooled sensitivity and specificity, and their accuracy was compared with that of radiologists. The present systematic review was registered in PROSPERO (CRD42019134460). RESULTS Nineteen studies with 4,781 thyroid nodules were included. Both the classic machine learning- and the deep learning-based CAD system had good performance in diagnosing malignant thyroid nodules (classic machine learning: sensitivity 0.86 [95% CI 0.79-0.92], specificity 0.85 [95% CI 0.77-0.91], diagnostic odds ratio (DOR) 37.41 [95% CI 24.91-56.20]; deep learning: sensitivity 0.89 [95% CI 0.81-0.93], specificity 0.84 [95% CI 0.75-0.90], DOR 40.87 [95% CI 18.13-92.13]). The diagnostic performance of the deep learning-based CAD system was comparable to that of the radiologists (sensitivity 0.87 [95% CI 0.78-0.93] vs. 0.87 [95% CI 0.85-0.89], specificity 0.85 [95% CI 0.76-0.91] vs. 0.87 [95% CI 0.81-0.91], DOR 40.12 [95% CI 15.58-103.33] vs. DOR 44.88 [95% CI 30.71-65.57]). CONCLUSIONS The CAD systems demonstrated good performance in diagnosing malignant thyroid nodules. However, experienced radiologists may still have an advantage over CAD systems during real-time diagnosis.
Collapse
Affiliation(s)
- Lei Xu
- Key Laboratory of Biomedical Information Engineering of the Ministry of Education, Department of Biomedical Engineering, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, China
- Xi'an Hospital of Traditional Chinese Medicine, Xi'an, China
| | - Junling Gao
- Key Laboratory of Biomedical Information Engineering of the Ministry of Education, Department of Biomedical Engineering, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, China
| | - Quan Wang
- Laboratory of Surgical Oncology, Peking University People's Hospital, Peking University, Beijing, China
| | - Jichao Yin
- Xi'an Hospital of Traditional Chinese Medicine, Xi'an, China
| | - Pengfei Yu
- Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Bin Bai
- Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Ruixia Pei
- Xi'an Hospital of Traditional Chinese Medicine, Xi'an, China
| | - Dingzhang Chen
- Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Guochun Yang
- Xi'an Hospital of Traditional Chinese Medicine, Xi'an, China
| | - Shiqi Wang
- Xijing Hospital, Fourth Military Medical University, Xi'an, China
- **Shiqi Wang, Xijing Hospital, Fourth Military Medical University, Changlexi St. 127, Xi'an 710032 (China), E-Mail
| | - Mingxi Wan
- Key Laboratory of Biomedical Information Engineering of the Ministry of Education, Department of Biomedical Engineering, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, China
- *Mingxi Wan, Key Laboratory of Biomedical Information Engineering of the Ministry of Education, Department of Biomedical Engineering, School of Life Science and Technology, Xi'an Jiaotong University, Xianningxi St. 28, Xi'an 710049 (China), E-Mail
| |
Collapse
|
35
|
False-Positive Malignant Diagnosis of Nodule Mimicking Lesions by Computer-Aided Thyroid Nodule Analysis in Clinical Ultrasonography Practice. Diagnostics (Basel) 2020; 10:diagnostics10060378. [PMID: 32517227 PMCID: PMC7345888 DOI: 10.3390/diagnostics10060378] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Revised: 06/03/2020] [Accepted: 06/04/2020] [Indexed: 11/17/2022] Open
Abstract
This study aims to test computer-aided diagnosis (CAD) for thyroid nodules in clinical ultrasonography (US) practice with a focus towards identifying thyroid entities associated with CAD system misdiagnoses. Two-hundred patients referred to thyroid US were prospectively enrolled. An experienced radiologist evaluated the thyroid nodules and saved axial images for further offline blinded analysis using a commercially available CAD system. To represent clinical practice, not only true nodules, but mimicking lesions were also included. Fine needle aspiration biopsy (FNAB) was performed according to present guidelines. US features and thyroid entities significantly associated with CAD system misdiagnosis were identified along with the diagnostic accuracy of the radiologist and the CAD system. Diagnostic specificity regarding the radiologist was significantly (p < 0.05) higher than when compared with the CAD system (88.1% vs. 40.5%) while no significant difference was found in the sensitivity (88.6% vs. 80%). Focal inhomogeneities and true nodules in thyroiditis, nodules with coarse calcification and inspissated colloid cystic nodules were significantly (p < 0.05) associated with CAD system misdiagnosis as false-positives. The commercially available CAD system is promising when used to exclude thyroid malignancies, however, it currently may not be able to reduce unnecessary FNABs, mainly due to the false-positive diagnoses of nodule mimicking lesions.
Collapse
|
36
|
Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners. J Ultrasound 2020; 23:169-174. [PMID: 32246401 DOI: 10.1007/s40477-020-00453-y] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2019] [Accepted: 03/13/2020] [Indexed: 12/15/2022] Open
Abstract
PURPOSE Computer-aided diagnosis (CAD) may improve interobserver agreement in the risk stratification of thyroid nodules. This study aims to evaluate the performance of the Korean Thyroid Imaging Reporting and Data System (K-TIRADS) classification as estimated by an expert radiologist, a senior resident, a medical student, and a CAD system, as well as the interobserver agreement among them. METHODS Between July 2016 and 2018, 107 nodules (size 5-40 mm, 27 malignant) were classified according to the K-TIRADS by an expert radiologist and CAD software. A third-year resident and a medical student with basic imaging training, both blinded to previous findings, retrospectively estimated the K-TIRADS classification. The diagnostic performance was calculated, including sensitivity, specificity, positive and negative predictive values, and the area under the receiver operating characteristic curve. RESULTS The CAD system and the expert achieved a sensitivity of 70.37% (95% CI 49.82-86.25%) and 81.48% (61.92-93.7%) and a specificity of 87.50% (78.21-93.84%) and 88.75% (79.72-94.72%), respectively. The specificity of the student was significantly lower (76.25% [65.42-85.05%], p = 0.02). CONCLUSION In our opinion, the CAD evaluation of thyroid nodules stratification risk has a potential role in a didactic field and does not play a real and effective role in the clinical field, where not only images but also specialistic medical practice is fundamental to achieve a diagnosis based on family history, genetics, lab tests, and so on. The CAD system may be useful for less experienced operators as its specificity was significantly higher.
Collapse
|
37
|
Lee JH, Ha EJ, Kim D, Jung YJ, Heo S, Jang YH, An SH, Lee K. Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT: external validation and clinical utility for resident training. Eur Radiol 2020; 30:3066-3072. [PMID: 32065285 DOI: 10.1007/s00330-019-06652-4] [Citation(s) in RCA: 45] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2019] [Revised: 12/17/2019] [Accepted: 12/19/2019] [Indexed: 12/12/2022]
Abstract
PURPOSE This study aimed to validate a deep learning model's diagnostic performance in using computed tomography (CT) to diagnose cervical lymph node metastasis (LNM) from thyroid cancer in a large clinical cohort and to evaluate the model's clinical utility for resident training. METHODS The performance of eight deep learning models was validated using 3838 axial CT images from 698 consecutive patients with thyroid cancer who underwent preoperative CT imaging between January and August 2018 (3606 and 232 images from benign and malignant lymph nodes, respectively). Six trainees viewed the same patient images (n = 242), and their diagnostic performance and confidence level (5-point scale) were assessed before and after computer-aided diagnosis (CAD) was included. RESULTS The overall area under the receiver operating characteristics (AUROC) of the eight deep learning algorithms was 0.846 (range 0.784-0.884). The best performing model was Xception, with an AUROC of 0.884. The diagnostic accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of Xception were 82.8%, 80.2%, 83.0%, 83.0%, and 80.2%, respectively. After introducing the CAD system, underperforming trainees received more help from artificial intelligence than the higher performing trainees (p = 0.046), and overall confidence levels significantly increased from 3.90 to 4.30 (p < 0.001). CONCLUSION The deep learning-based CAD system used in this study for CT diagnosis of cervical LNM from thyroid cancer was clinically validated with an AUROC of 0.884. This approach may serve as a training tool to help resident physicians to gain confidence in diagnosis. KEY POINTS • A deep learning-based CAD system for CT diagnosis of cervical LNM from thyroid cancer was validated using data from a clinical cohort. The AUROC for the eight tested algorithms ranged from 0.784 to 0.884. • Of the eight models, the Xception algorithm was the best performing model for the external validation dataset with 0.884 AUROC. The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 82.8%, 80.2%, 83.0%, 83.0%, and 80.2%, respectively. • The CAD system exhibited potential to improve diagnostic specificity and accuracy in underperforming trainees (3 of 6 trainees, 50.0%). This approach may have clinical utility as a training tool to help trainees to gain confidence in diagnoses.
Collapse
Affiliation(s)
- Jeong Hoon Lee
- Division of Biomedical Informatics, Seoul National University Biomedical Informatics (SNUBI), Seoul National University College of Medicine, Seoul, 110799, South Korea
| | - Eun Ju Ha
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea.
| | - DaYoung Kim
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| | - Yong Jun Jung
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| | - Subin Heo
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| | - Yong-Ho Jang
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| | - Sung Hyun An
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| | - Kyungmin Lee
- Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea
| |
Collapse
|
38
|
Barczyński M, Stopa-Barczyńska M, Wojtczak B, Czarniecka A, Konturek A. Clinical validation of S-Detect TM mode in semi-automated ultrasound classification of thyroid lesions in surgical office. Gland Surg 2020; 9:S77-S85. [PMID: 32175248 DOI: 10.21037/gs.2019.12.23] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Background In recent years well-recognized scientific societies introduced guidelines for ultrasound (US) malignancy risk stratification of thyroid nodules. These guidelines categorize the risk of malignancy in relation to a combination of several US features. Based on these US image lexicons an US-based computer-aided diagnosis (CAD) systems were developed. Nevertheless, their clinical utility has not been evaluated in any study of surgeon-performed office US of the thyroid. Hence, the aim of this pilot study was to validate s-DetectTM mode in semi-automated US classification of thyroid lesions during surgeon-performed office US. Methods This is a prospective study of 50 patients who underwent surgeon-performed thyroid US (basic US skills without CAD vs. with CAD vs. expert US skills without CAD) in the out-patient office as part of the preoperative workup. The real-time CAD system software using artificial intelligence (S-DetectTM for Thyroid; Samsung Medison Co.) was integrated into the RS85 US system. Primary outcome was CAD system added-value to the surgeon-performed office US evaluation. Secondary outcomes were: diagnostic accuracy of CAD system, intra and interobserver variability in the US assessment of thyroid nodules. Surgical pathology report was used to validate the pre-surgical diagnosis. Results CAD system added-value to thyroid assessment by a surgeon with basic US skills was equal to 6% (overall accuracy of 82% for evaluation with CAD vs. 76% for evaluation without CAD system; P<0.001), and final diagnosis was different than predicted by US assessment in 3 patients (1 more true-positive and 2 more true-negative results). However, CAD system was inferior to thyroid assessment by a surgeon with expert US skills in 6 patients who had false-positive results (P<0.001). Conclusions The sensitivity and negative predictive value of CAD system for US classification of thyroid lesions were similar as surgeon with expert US skills whereas specificity and positive predictive value were significantly inferior but markedly better than judgement of a surgeon with basic US skills alone.
Collapse
Affiliation(s)
- Marcin Barczyński
- Department of Endocrine Surgery, Third Chair of General Surgery, Jagiellonian University Medical College, Kraków, Poland
| | - Małgorzata Stopa-Barczyńska
- Clinical Ward of General Surgery and Oncology, Gabriel Narutowicz Memorial Municipal Hospital, Kraków, Poland
| | - Beata Wojtczak
- Department of General, Minimally Invasive and Endocrine Surgery, Wroclaw Medical University, Wroclaw, Poland
| | - Agnieszka Czarniecka
- Department of Oncological and Reconstructive Surgery, M. Sklodowska-Curie Institute - Oncology Centre, Gliwice, Poland
| | - Aleksander Konturek
- Department of Endocrine Surgery, Third Chair of General Surgery, Jagiellonian University Medical College, Kraków, Poland
| |
Collapse
|