Combining many-objective radiomics and 3D convolutional neural network through evidential reasoning to predict lymph node metastasis in head and neck cancer

Artificial Intelligence-Driven Radiomics in Head and Neck Cancer: Current Status and Future Prospects

BACKGROUND

Radiomics is a rapidly growing field used to leverage medical radiological images by extracting quantitative features. These are supposed to characterize a patient's phenotype, and when combined with artificial intelligence techniques, to improve the accuracy of diagnostic models and clinical outcome prediction.

OBJECTIVES

This review aims at examining the application areas of artificial intelligence-based radiomics (AI-based radiomics) for the management of head and neck cancer (HNC). It further explores the workflow of AI-based radiomics for personalized and precision oncology in HNC. Finally, it examines the current challenges of AI-based radiomics in daily clinical oncology and offers possible solutions to these challenges.

METHODS

Comprehensive electronic databases (PubMed, Medline via Ovid, Scopus, Web of Science, CINAHL, and Cochrane Library) were searched following the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines. The quality of included studies and their risk of biases were evaluated using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD)and Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

Out of the 659 search hits retrieved, 45 fulfilled the inclusion criteria. Our review revealed that the application of AI-based radiomics model as an ancillary tool for improved decision-making in HNC management includes radiomics-based cancer diagnosis and radiomics-based cancer prognosis. The radiomics-based cancer diagnosis includes tumor staging, tumor grading, and classification of malignant and benign tumors. Similarly, radiomics-based cancer prognosis includes prediction for treatment response, recurrence, metastasis, and survival. In addition, the challenges in the implementation of these models for clinical evaluations include data imbalance, feature engineering (extraction and selection), model generalizability, multi-modal fusion, and model interpretability.

CONCLUSION

Considering the highly subjective and interobserver variability that is peculiar to the interpretation of medical images by expert clinicians, AI-based radiomics seeks to offer potentially useful quantitative information, which is not visible to the human eye or unintentionally often remain ignored during clinical imaging practice. By enabling the extraction of this type of information, AI-based radiomics has the potential to revolutionize HNC oncology, providing a platform for more personalized, higher quality, and cost-effective care for HNC patients.

A gradient mapping guided explainable deep neural network for extracapsular extension identification in 3D head and neck cancer computed tomography images

BACKGROUND

Diagnosis and treatment management for head and neck squamous cell carcinoma (HNSCC) is guided by routine diagnostic head and neck computed tomography (CT) scans to identify tumor and lymph node features. The extracapsular extension (ECE) is a strong predictor of patients' survival outcomes with HNSCC. It is essential to detect the occurrence of ECE as it changes staging and treatment planning for patients. Current clinical ECE detection relies on visual identification and pathologic confirmation conducted by clinicians. However, manual annotation of the lymph node region is a required data preprocessing step in most of the current machine learning-based ECE diagnosis studies.

PURPOSE

In this paper, we propose a Gradient Mapping Guided Explainable Network (GMGENet) framework to perform ECE identification automatically without requiring annotated lymph node region information.

METHODS

The gradient-weighted class activation mapping (Grad-CAM) technique is applied to guide the deep learning algorithm to focus on the regions that are highly related to ECE. The proposed framework includes an extractor and a classifier. In a joint training process, informative volumes of interest (VOIs) are extracted by the extractor without labeled lymph node region information, and the classifier learns the pattern to classify the extracted VOIs into ECE positive and negative.

RESULTS

In evaluation, the proposed methods are well-trained and tested using cross-validation. GMGENet achieved test accuracy and area under the curve (AUC) of 92.2% and 89.3%, respectively. GMGENetV2 achieved 90.3% accuracy and 91.7% AUC in the test. The results were compared with different existing models and further confirmed and explained by generating ECE probability heatmaps via a Grad-CAM technique. The presence or absence of ECE has been analyzed and correlated with ground truth histopathological findings.

CONCLUSIONS

The proposed deep network can learn meaningful patterns to identify ECE without providing lymph node contours. The introduced ECE heatmaps will contribute to the clinical implementations of the proposed model and reveal unknown features to radiologists. The outcome of this study is expected to promote the implementation of explainable artificial intelligence-assiste ECE detection.

Comparing three-dimensional and two-dimensional deep-learning, radiomics, and fusion models for predicting occult lymph node metastasis in laryngeal squamous cell carcinoma based on CT imaging: a multicentre, retrospective, diagnostic study

Background

The occult lymph node metastasis (LNM) of laryngeal squamous cell carcinoma (LSCC) affects the treatment and prognosis of patients. This study aimed to comprehensively compare the performance of the three-dimensional and two-dimensional deep learning models, radiomics model, and the fusion models for predicting occult LNM in LSCC.

Methods

In this retrospective diagnostic study, a total of 553 patients with clinical N0 stage LSCC, who underwent surgical treatment without distant metastasis and multiple primary cancers, were consecutively enrolled from four Chinese medical centres between January 01, 2016 and December 30, 2020. The participant data were manually retrieved from medical records, imaging databases, and pathology reports. The study cohort was divided into a training set (n = 300), an internal test set (n = 89), and two external test sets (n = 120 and 44, respectively). The three-dimensional deep learning (3D DL), two-dimensional deep learning (2D DL), and radiomics model were developed using CT images of the primary tumor. The clinical model was constructed based on clinical and radiological features. Two fusion strategies were utilized to develop the fusion model: the feature-based DLRad_FB model and the decision-based DLRad_DB model. The discriminative ability and correlation of 3D DL, 2D DL and radiomics features were analysed comprehensively. The performances of the predictive models were evaluated based on the pathological diagnosis.

Findings

The 3D DL features had superior discriminative ability and lower internal redundancy compared to 2D DL and radiomics features. The DLRad_DB model achieved the highest AUC (0.89-0.90) among all the study sets, significantly outperforming the clinical model (AUC = 0.73-0.78, P = 0.0001-0.042, Delong test). Compared to the DLRad_DB model, the AUC values for the DLRad_FB, 3D DL, 2D DL, and radiomics models were 0.82-0.84 (P = 0.025-0.46), 0.86-0.89 (P = 0.75-0.97), 0.83-0.86 (P = 0.029-0.66), and 0.79-0.82 (P = 0.0072-0.10), respectively in the study sets. Additionally, the DLRad_DB model exhibited the best sensitivity (82-88%) and specificity (79-85%) in the test sets.

Interpretation

The decision-based fusion model DLRad_DB, which combines 3D DL, 2D DL, radiomics, and clinical data, can be utilized to predict occult LNM in LSCC. This has the potential to minimize unnecessary lymph node dissection and prophylactic radiotherapy in patients with cN0 disease.

Funding

National Natural Science Foundation of China, Natural Science Foundation of Shandong Province.

Convolutional neural network-based program to predict lymph node metastasis of non-small cell lung cancer using 18F-FDG PET

PURPOSE

To develop a convolutional neural network (CNN)-based program to analyze maximum intensity projection (MIP) images of 2-deoxy-2-[F-18]fluoro-D-glucose (FDG) positron emission tomography (PET) scans, aimed at predicting lymph node metastasis of non-small cell lung cancer (NSCLC), and to evaluate its effectiveness in providing diagnostic assistance to radiologists.

METHODS

We obtained PET images of NSCLC from public datasets, including those of 435 patients with available N-stage information, which were divided into a training set (n = 304) and a test set (n = 131). We generated 36 maximum intensity projection (MIP) images for each patient. A residual network (ResNet-50)-based CNN was trained using the MIP images of the training set to predict lymph node metastasis. Lymph node metastasis in the test set was predicted by the trained CNN as well as by seven radiologists twice: first without and second with CNN assistance. Diagnostic performance metrics, including accuracy and prediction error (the difference between the truth and the predictions), were calculated, and reading times were recorded.

RESULTS

In the test set, 67 (51%) patients exhibited lymph node metastases and the CNN yielded 0.748 predictive accuracy. With the assistance of the CNN, the prediction error was significantly reduced for six of the seven radiologists although the accuracy did not change significantly. The prediction time was significantly reduced for five of the seven radiologists with the median reduction ratio 38.0%.

CONCLUSION

The CNN-based program could potentially assist radiologists in predicting lymph node metastasis by increasing diagnostic confidence and reducing reading time without affecting diagnostic accuracy, at least in the limited situations using MIP images.

Review of artificial intelligence clinical applications in Nuclear Medicine

This paper provides an in-depth analysis of the clinical applications of artificial intelligence (AI) in Nuclear Medicine, focusing on three key areas: neurology, cardiology, and oncology. Beginning with neurology, specifically Alzheimer's disease and Parkinson's disease, the paper examines reviews on diagnosis and treatment planning. The same pattern is followed in cardiology studies. In the final section on oncology, the paper explores the various AI applications in multiple cancer types, including lung, head and neck, lymphoma, and pancreatic cancer.

The Use of Artificial Intelligence Algorithms in the Prognosis and Detection of Lymph Node Involvement in Head and Neck Cancer and Possible Impact in the Development of Personalized Therapeutic Strategy: A Systematic Review

Given the increasingly important role that the use of artificial intelligence algorithms is taking on in the medical field today (especially in oncology), the purpose of this systematic review is to analyze the main reports on such algorithms applied for the prognostic evaluation of patients with head and neck malignancies. The objective of this paper is to examine the currently available literature in the field of artificial intelligence applied to head and neck oncology, particularly in the prognostic evaluation of the patient with this kind of tumor, by means of a systematic review. The paper exposes an overview of the applications of artificial intelligence in deriving prognostic information related to the prediction of survival and recurrence and how these data may have a potential impact on the choice of therapeutic strategy, making it increasingly personalized. This systematic review was written following the PRISMA 2020 guidelines.

Integrating Artificial Intelligence and Machine Learning Into Cancer Clinical Trials

The practice of oncology requires analyzing and synthesizing abundant data. From the patient's workup to determine eligibility to the therapies received to the post-treatment surveillance, practitioners must constantly juggle, evaluate, and weigh decision-making based on their best understanding of information at hand. These complex, multifactorial decisions have a tremendous opportunity to benefit from data-driven machine learning (ML) methods to drive opportunities in artificial intelligence (AI). Within the past 5 years, we have seen AI move from simply a promising opportunity to being used in prospective trials. Here, we review recent efforts of AI in clinical trials that have moved the needle towards improved prediction of actionable outcomes, such as predicting acute care visits, short term mortality, and pathologic extranodal extension. We then pause and reflect on how these AI models ask a different question than traditional statistics models that readers may be more familiar with; how then should readers conceptualize and interpret AI models that they are not as familiar with. We end with what we believe are promising future opportunities for AI in oncology, with an eye towards allowing the data to inform us through unsupervised learning and generative models, rather than asking AI to perform specific functions.

Acute coronary event (ACE) prediction following breast radiotherapy by features extracted from 3D CT, dose, and cardiac structures

PURPOSE

Heart toxicity, such as major acute coronary events (ACE), following breast radiation therapy (RT) is of utmost concern. Thus, many studies have been investigating the effect of mean heart dose (MHD) and dose received in heart sub-structures on toxicity. Most studies focused on the dose thresholds in the heart and its sub-structures, while few studies adopted such computational methods as deep neural networks (DNN) and radiomics. This work aims to construct a feature-driven predictive model for ACE after breast RT.

METHODS

A recently proposed two-step predictive model that extracts a number of features from a deep auto-segmentation network and processes the selected features for prediction was adopted. This work refined the auto-segmenting network and feature processing algorithms to enhance performance in cardiac toxicity prediction. In the predictive model, the deep convolutional neural network (CNN) extracted features from 3D computed tomography (CT) images and dose distributions in three automatically segmented heart sub-structures, including the left anterior descending artery (LAD), right coronary artery (RCA), and left ventricle (LV). The optimal feature processing workflow for the extracted features was explored to enhance the prediction accuracy. The regions associated with toxicity were visualized using a class activation map (CAM)-based technique. Our proposed model was validated against a conventional DNN (convolutional and fully connected layers) and radiomics with a patient cohort of 84 cases, including 29 and 55 patient cases with and without ACE. Of the entire 84 cases, 12 randomly chosen cases (5 toxicity and 7 non-toxicity cases) were set aside for independent test, and the remaining 72 cases were applied to 4-fold stratified cross-validation.

RESULTS

Our predictive model outperformed the conventional DNN by 38% and 10% and radiomics-based predictive models by 9% and 10% in AUC for 4-fold cross-validations and independent test, respectively. The degree of enhancement was greater when incorporating dose information and heart sub-structures into feature extraction. The model whose inputs were CT, dose, and three sub-structures (LV, LAD, and RCA) reached 96% prediction accuracy on average and 0.94 area under the curve (AUC) on average in the cross-validation, and also achieved prediction accuracy of 83% and AUC of 0.83 in the independent test. On 10 correctly predicted cases out of 12 for the independent test, the activation maps implied that for cases of ACE toxicity, the higher intensity was more likely to be observed inside the LV.

CONCLUSIONS

The proposed model characterized by modifications in model input with dose distributions and cardiac sub-structures, and serial processing of feature extraction and feature selection techniques can improve the predictive performance in ACE following breast RT.

Efficacy and Quality-of-Life Following Involved Nodal Radiotherapy for Head and Neck Squamous Cell Carcinoma: The INRT-AIR Phase II Clinical Trial

PURPOSE

Elective neck irradiation (ENI) has long been considered mandatory when treating head and neck squamous cell carcinoma (HNSCC) with definitive radiotherapy, but it is associated with significant dose to normal organs-at-risk (OAR). In this prospective phase II study, we investigated the efficacy and tolerability of eliminating ENI and strictly treating involved and suspicious lymph nodes (LN) with intensity-modulated radiotherapy.

PATIENTS AND METHODS

Patients with newly diagnosed HNSCC of the oropharynx, larynx, and hypopharynx were eligible for enrollment. Each LN was characterized as involved or suspicious based on radiologic criteria and an in-house artificial intelligence (AI)-based classification model. Gross disease received 70 Gray (Gy) in 35 fractions and suspicious LNs were treated with 66.5 Gy, without ENI. The primary endpoint was solitary elective volume recurrence, with secondary endpoints including patterns-of-failure and patient-reported outcomes.

RESULTS

Sixty-seven patients were enrolled, with 18 larynx/hypopharynx and 49 oropharynx cancer. With a median follow-up of 33.4 months, the 2-year risk of solitary elective nodal recurrence was 0%. Gastrostomy tubes were placed in 14 (21%), with median removal after 2.9 months for disease-free patients; no disease-free patient is chronically dependent. Grade I/II dermatitis was seen in 90%/10%. There was no significant decline in composite MD Anderson Dysphagia Index scores after treatment, with means of 89.1 and 92.6 at 12 and 24 months, respectively.

CONCLUSIONS

These results suggest that eliminating ENI is oncologically sound for HNSCC, with highly favorable quality-of-life outcomes. Additional prospective studies are needed to support this promising paradigm before implementation in any nontrial setting.

Automated localization and segmentation of cervical lymph nodes on contrast-enhanced CT using a 3D foveal fully convolutional neural network

BACKGROUND

In the management of cancer patients, determination of TNM status is essential for treatment decision-making and therefore closely linked to clinical outcome and survival. Here, we developed a tool for automatic three-dimensional (3D) localization and segmentation of cervical lymph nodes (LNs) on contrast-enhanced computed tomography (CECT) examinations.

METHODS

In this IRB-approved retrospective single-center study, 187 CECT examinations of the head and neck region from patients with various primary diseases were collected from our local database, and 3656 LNs (19.5 ± 14.9 LNs/CECT, mean ± standard deviation) with a short-axis diameter (SAD) ≥ 5 mm were segmented manually by expert physicians. With these data, we trained an independent fully convolutional neural network based on 3D foveal patches. Testing was performed on 30 independent CECTs with 925 segmented LNs with an SAD ≥ 5 mm.

RESULTS

In total, 4,581 LNs were segmented in 217 CECTs. The model achieved an average localization rate (LR), i.e., percentage of localized LNs/CECT, of 78.0% in the validation dataset. In the test dataset, average LR was 81.1% with a mean Dice coefficient of 0.71. For enlarged LNs with a SAD ≥ 10 mm, LR was 96.2%. In the test dataset, the false-positive rate was 2.4 LNs/CECT.

CONCLUSIONS

Our trained AI model demonstrated a good overall performance in the consistent automatic localization and 3D segmentation of physiological and metastatic cervical LNs with a SAD ≥ 5 mm on CECTs. This could aid clinical localization and automatic 3D segmentation, which can benefit clinical care and radiomics research.

RELEVANCE STATEMENT

Our AI model is a time-saving tool for 3D segmentation of cervical lymph nodes on contrast-enhanced CT scans and serves as a solid base for N staging in clinical practice and further radiomics research.

KEY POINTS

• Determination of N status in TNM staging is essential for therapy planning in oncology. • Segmenting cervical lymph nodes manually is highly time-consuming in clinical practice. • Our model provides a robust, automated 3D segmentation of cervical lymph nodes. • It achieves a high accuracy for localization especially of enlarged lymph nodes. • These segmentations should assist clinical care and radiomics research.

Imaging of Neck Nodes in Head and Neck Cancers - a Comprehensive Update

Cervical lymph node metastases from head and neck squamous cell cancers significantly reduce disease-free survival and worsen overall prognosis and, hence, deserve more aggressive management and follow-up. As per the eighth edition of the American Joint Committee on Cancer staging manual, extranodal extension, especially in human papillomavirus-negative cancers, has been incorporated in staging as it is important in deciding management and significantly impacts the outcome of head and neck squamous cell cancer. Lymph node imaging with various radiological modalities, including ultrasound, computed tomography and magnetic resonance imaging, has been widely used, not only to demonstrate nodal involvement but also for guided histopathological evaluation and therapeutic intervention. Computed tomography and magnetic resonance imaging, together with positron emission tomography, are used widely for the follow-up of treated patients. Finally, there is an emerging role for artificial intelligence in neck node imaging that has shown promising results, increasing the accuracy of detection of nodal involvement, especially normal-appearing nodes. The aim of this review is to provide a comprehensive overview of the diagnosis and management of involved neck nodes with a focus on sentinel node anatomy, pathogenesis, imaging correlates (including radiogenomics and artificial intelligence) and the role of image-guided interventions.

The Promise and Future of Radiomics for Personalized Radiotherapy Dosing and Adaptation

Quantitative image analysis, also known as radiomics, aims to analyze large-scale quantitative features extracted from acquired medical images using hand-crafted or machine-engineered feature extraction approaches. Radiomics has great potential for a variety of clinical applications in radiation oncology, an image-rich treatment modality that utilizes computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) for treatment planning, dose calculation, and image guidance. A promising application of radiomics is in predicting treatment outcomes after radiotherapy such as local control and treatment-related toxicity using features extracted from pretreatment and on-treatment images. Based on these individualized predictions of treatment outcomes, radiotherapy dose can be sculpted to meet the specific needs and preferences of each patient. Radiomics can aid in tumor characterization for personalized targeting, especially for identifying high-risk regions within a tumor that cannot be easily discerned based on size or intensity alone. Radiomics-based treatment response prediction can aid in developing personalized fractionation and dose adjustments. In order to make radiomics models more applicable across different institutions with varying scanners and patient populations, further efforts are needed to harmonize and standardize the acquisition protocols by minimizing uncertainties within the imaging data.

Attention-based Deep Learning for the Preoperative Differentiation of Axillary Lymph Node Metastasis in Breast Cancer on DCE-MRI

BACKGROUND

Previous studies have explored the potential on radiomics features of primary breast cancer tumor to identify axillary lymph node (ALN) metastasis. However, the value of deep learning (DL) to identify ALN metastasis remains unclear.

PURPOSE

To investigate the potential of the proposed attention-based DL model for the preoperative differentiation of ALN metastasis in breast cancer on dynamic contrast-enhanced MRI (DCE-MRI).

STUDY TYPE

Retrospective.

POPULATION

A total of 941 breast cancer patients who underwent DCE-MRI before surgery were included in the training (742 patients), internal test (83 patients), and external test (116 patients) cohorts.

FIELD STRENGTH/SEQUENCE

A 3.0 T MR scanner, DCE-MRI sequence.

ASSESSMENT

A DL model containing a 3D deep residual network (ResNet) architecture and a convolutional block attention module, named RCNet, was proposed for ALN metastasis identification. Three RCNet models were established based on the tumor, ALN, and combined tumor-ALN regions on the images. The performance of these models was compared with ResNet models, radiomics models, the Memorial Sloan-Kettering Cancer Center (MSKCC) model, and three radiologists (W.L., H.S., and F. L.).

STATISTICAL TESTS

Dice similarity coefficient for breast tumor and ALN segmentation. Accuracy, sensitivity, specificity, intercorrelation and intracorrelation coefficients, area under the curve (AUC), and Delong test for ALN classification.

RESULTS

The optimal RCNet model, that is, RCNet-tumor+ALN , achieved an AUC of 0.907, an accuracy of 0.831, a sensitivity of 0.824, and a specificity of 0.837 in the internal test cohort, as well as an AUC of 0.852, an accuracy of 0.828, a sensitivity of 0.792, and a specificity of 0.853 in the external test cohort. Additionally, with the assistance of RCNet-tumor+ALN , the radiologists' performance was improved (external test cohort, P < 0.05).

DATA CONCLUSION

DCE-MRI-based RCNet model could provide a noninvasive auxiliary tool to identify ALN metastasis preoperatively in breast cancer, which may assist radiologists in conducting more accurate evaluation of ALN status.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Artificial intelligence-driven radiomics study in cancer: the role of feature engineering and modeling

Modern medicine is reliant on various medical imaging technologies for non-invasively observing patients' anatomy. However, the interpretation of medical images can be highly subjective and dependent on the expertise of clinicians. Moreover, some potentially useful quantitative information in medical images, especially that which is not visible to the naked eye, is often ignored during clinical practice. In contrast, radiomics performs high-throughput feature extraction from medical images, which enables quantitative analysis of medical images and prediction of various clinical endpoints. Studies have reported that radiomics exhibits promising performance in diagnosis and predicting treatment responses and prognosis, demonstrating its potential to be a non-invasive auxiliary tool for personalized medicine. However, radiomics remains in a developmental phase as numerous technical challenges have yet to be solved, especially in feature engineering and statistical modeling. In this review, we introduce the current utility of radiomics by summarizing research on its application in the diagnosis, prognosis, and prediction of treatment responses in patients with cancer. We focus on machine learning approaches, for feature extraction and selection during feature engineering and for imbalanced datasets and multi-modality fusion during statistical modeling. Furthermore, we introduce the stability, reproducibility, and interpretability of features, and the generalizability and interpretability of models. Finally, we offer possible solutions to current challenges in radiomics research.

ARMO: automated and reliable multi-objective model for lymph node metastasis prediction in head and neck cancer

Objective. Accurate diagnosis of lymph node metastasis (LNM) is critical in treatment management for patients with head and neck cancer. Positron emission tomography and computed tomography are routinely used for identifying LNM status. However, for small or less fluorodeoxyglucose (FDG) avid nodes, there are always uncertainties in LNM diagnosis. We are aiming to develop a reliable prediction model is for identifying LNM.Approach. In this study, a new automated and reliable multi-objective learning model (ARMO) is proposed. In ARMO, a multi-objective model is introduced to obtain balanced sensitivity and specificity. Meanwhile, confidence is calibrated by introducing individual reliability, whilst the model uncertainty is estimated by a newly defined overall reliability in ARMO. In the training stage, a Pareto-optimal model set is generated. Then all the Pareto-optimal models are used, and a reliable fusion strategy that introduces individual reliability is developed for calibrating the confidence of each output. The overall reliability is calculated to estimate the model uncertainty for each test sample.Main results. The experimental results demonstrated that ARMO obtained more promising results, which the area under the curve, accuracy, sensitivity and specificity can achieve 0.97, 0.93, 0.88 and 0.94, respectively. Meanwhile, based on calibrated confidence and overall reliability, clinicians could pay particular attention to highly uncertain predictions.Significance. In this study, we developed a unified model that can achieve balanced prediction, confidence calibration and uncertainty estimation simultaneously. The experimental results demonstrated that ARMO can obtain accurate and reliable prediction performance.

Towards reliable head and neck cancers locoregional recurrence prediction using delta-radiomics and learning with rejection option

PURPOSE

A reliable locoregional recurrence (LRR) prediction model is important for the personalized management of head and neck cancers (HNC) patients who received radiotherapy. This work aims to develop a delta-radiomics feature-based multi-classifier, multi-objective, and multi-modality (Delta-mCOM) model for post-treatment HNC LRR prediction. Furthermore, we aim to adopt a learning with rejection option (LRO) strategy to boost the reliability of Delta-mCOM model by rejecting prediction for samples with high prediction uncertainties.

METHODS

In this retrospective study, we collected PET/CT image and clinical data from 224 HNC patients who received radiotherapy (RT) at our institution. We calculated the differences between radiomics features extracted from PET/CT images acquired before and after radiotherapy and used them in conjunction with pre-treatment radiomics features as the input features. Using clinical parameters, PET radiomics features, and CT radiomics features, we built and optimized three separate single-modality models. We used multiple classifiers for model construction and employed sensitivity and specificity simultaneously as the training objectives for each of them. Then, for testing samples, we fused the output probabilities from all these single-modality models to obtain the final output probabilities of the Delta-mCOM model. In the LRO strategy, we estimated the epistemic and aleatoric uncertainties when predicting with a trained Delta-mCOM model and identified patients associated with prediction of higher reliability (low uncertainty estimates). The epistemic and aleatoric uncertainties were estimated using an AutoEncoder-style anomaly detection model and test-time augmentation (TTA) with predictions made from the Delta-mCOM model, respectively. Predictions with higher epistemic uncertainty or higher aleatoric uncertainty than given thresholds were deemed unreliable, and they were rejected before providing a final prediction. In this study, different thresholds corresponding to different low-reliability prediction rejection ratios were applied. Their values are based on the estimated epistemic and aleatoric uncertainties distribution of the validation data.

RESULTS

The Delta-mCOM model performed significantly better than the single-modality models, whether trained with pre-, post-treatment radiomics features or concatenated BaseLine and Delta-Radiomics Features (BL-DRFs). It was numerically superior to the PET and CT fused BL-DRF model (nonstatistically significant). Using the LRO strategy for the Delta-mCOM model, most of the evaluation metrics improved as the rejection ratio increased from 0% to around 25%. Utilizing both epistemic and aleatoric uncertainty for rejection yielded nonstatistically significant improved metrics compared to each alone at approximately a 25% rejection ratio. Metrics were significantly better than the no-rejection method when the reject ratio was higher than 50%.

CONCLUSIONS

The inclusion of the delta-radiomics feature improved the accuracy of HNC LRR prediction, and the proposed Delta-mCOM model can give more reliable predictions by rejecting predictions for samples of high uncertainty using the LRO strategy.

Radiomics analysis of [18F]-fluoro-2-deoxyglucose positron emission tomography for the prediction of cervical lymph node metastasis in tongue squamous cell carcinoma

OBJECTIVES

This study aimed to create a predictive model for cervical lymph node metastasis (CLNM) in patients with tongue squamous cell carcinoma (SCC) based on radiomics features detected by [¹⁸F]-fluoro-2-deoxyglucose (¹⁸F-FDG) positron emission tomography (PET).

METHODS

A total of 40 patients with tongue SCC who underwent ¹⁸F-FDG PET imaging during their first medical examination were enrolled. During the follow-up period (mean 28 months), 20 patients had CLNM, including six with late CLNM, whereas the remaining 20 patients did not have CLNM. Radiomics features were extracted from ¹⁸F-FDG PET images of all patients irrespective of metal artifact, and clinicopathological factors were obtained from the medical records. Late CLNM was defined as the CLNM that occurred after major treatment. The least absolute shrinkage and selection operator (LASSO) model was used for radiomics feature selection and sequential data fitting. The receiver operating characteristic curve analysis was used to assess the predictive performance of the ¹⁸F-FDG PET-based model and clinicopathological factors model (CFM) for CLNM.

RESULTS

Six radiomics features were selected from LASSO analysis. The average values of the area under the curve (AUC), accuracy, sensitivity, and specificity of radiomics analysis for predicting CLNM from ¹⁸F-FDG PET images were 0.79, 0.68, 0.65, and 0.70, respectively. In contrast, those of the CFM were 0.54, 0.60, 0.60, and 0.60, respectively. The ¹⁸F-FDG PET-based model showed significantly higher AUC than that of the CFM.

CONCLUSIONS

The ¹⁸F-FDG PET-based model has better potential for diagnosing CLNM and predicting late CLNM in patients with tongue SCC than the CFM.

Machine-learning based investigation of prognostic indicators for oncological outcome of pancreatic ductal adenocarcinoma

Introduction

Pancreatic ductal adenocarcinoma (PDAC) is an aggressive malignancy with a poor prognosis. Surgical resection remains the only potential curative treatment option for early-stage resectable PDAC. Patients with locally advanced or micrometastatic disease should ideally undergo neoadjuvant therapy prior to surgical resection for an optimal treatment outcome. Computerized tomography (CT) scan is the most common imaging modality obtained prior to surgery. However, the ability of CT scans to assess the nodal status and resectability remains suboptimal and depends heavily on physician experience. Improved preoperative radiographic tumor staging with the prediction of postoperative margin and the lymph node status could have important implications in treatment sequencing. This paper proposes a novel machine learning predictive model, utilizing a three-dimensional convoluted neural network (3D-CNN), to reliably predict the presence of lymph node metastasis and the postoperative positive margin status based on preoperative CT scans.

Methods

A total of 881 CT scans were obtained from 110 patients with PDAC. Patients and images were separated into training and validation groups for both lymph node and margin prediction studies. Per-scan analysis and per-patient analysis (utilizing majority voting method) were performed.

Results

For a lymph node prediction 3D-CNN model, accuracy was 90% for per-patient analysis and 75% for per-scan analysis. For a postoperative margin prediction 3D-CNN model, accuracy was 81% for per-patient analysis and 76% for per-scan analysis.

Discussion

This paper provides a proof of concept that utilizing radiomics and the 3D-CNN deep learning framework may be used preoperatively to improve the prediction of positive resection margins as well as the presence of lymph node metastatic disease. Further investigations should be performed with larger cohorts to increase the generalizability of this model; however, there is a great promise in the use of convoluted neural networks to assist clinicians with treatment selection for patients with PDAC.

Deep learning and radiomics to predict the mitotic index of gastrointestinal stromal tumors based on multiparametric MRI

Introduction

Preoperative evaluation of the mitotic index (MI) of gastrointestinal stromal tumors (GISTs) represents the basis of individualized treatment of patients. However, the accuracy of conventional preoperative imaging methods is limited. The aim of this study was to develop a predictive model based on multiparametric MRI for preoperative MI prediction.

Methods

A total of 112 patients who were pathologically diagnosed with GIST were enrolled in this study. The dataset was subdivided into the development (n = 81) and test (n = 31) sets based on the time of diagnosis. With the use of T2-weighted imaging (T2WI) and apparent diffusion coefficient (ADC) map, a convolutional neural network (CNN)-based classifier was developed for MI prediction, which used a hybrid approach based on 2D tumor images and radiomics features from 3D tumor shape. The trained model was tested on an internal test set. Then, the hybrid model was comprehensively tested and compared with the conventional ResNet, shape radiomics classifier, and age plus diameter classifier.

Results

The hybrid model showed good MI prediction ability at the image level; the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and accuracy in the test set were 0.947 (95% confidence interval [CI]: 0.927-0.968), 0.964 (95% CI: 0.930-0.978), and 90.8 (95% CI: 88.0-93.0), respectively. With the average probabilities from multiple samples per patient, good performance was also achieved at the patient level, with AUROC, AUPRC, and accuracy of 0.930 (95% CI: 0.828-1.000), 0.941 (95% CI: 0.792-1.000), and 93.6% (95% CI: 79.3-98.2) in the test set, respectively.

Discussion

The deep learning-based hybrid model demonstrated the potential to be a good tool for the operative and non-invasive prediction of MI in GIST patients.

Current Applications of Artificial Intelligence to Classify Cervical Lymph Nodes in Patients with Head and Neck Squamous Cell Carcinoma-A Systematic Review

Locally-advanced head and neck squamous cell carcinoma (HNSCC) is mainly defined by the presence of pathologic cervical lymph nodes (LNs) with or without extracapsular spread (ECS). Current radiologic criteria to classify LNs as non-pathologic, pathologic, or pathologic with ECS are primarily shape-based. However, significantly more quantitative information is contained within imaging modalities. This quantitative information could be exploited for classification of LNs in patients with locally-advanced HNSCC by means of artificial intelligence (AI). Currently, various reviews exploring the role of AI in HNSCC are available. However, reviews specifically addressing the current role of AI to classify LN in HNSCC-patients are sparse. The present work systematically reviews original articles that specifically explore the role of AI to classify LNs in locally-advanced HNSCC applying Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines and the Study Quality Assessment Tool of National Institute of Health (NIH). Between 2001 and 2022, out of 69 studies a total of 13 retrospective, mainly monocentric, studies were identified. The majority of the studies included patients with oropharyngeal and oral cavity (9 and 7 of 13 studies, respectively) HNSCC. Histopathologic findings were defined as reference in 9 of 13 studies. Machine learning was applied in 13 studies, 9 of them applying deep learning. The mean number of included patients was 75 (SD ± 72; range 10-258) and of LNs was 340 (SD ± 268; range 21-791). The mean diagnostic accuracy for the training sets was 86% (SD ± 14%; range: 43-99%) and for testing sets 86% (SD ± 5%; range 76-92%). Consequently, all of the identified studies concluded AI to be a potentially promising diagnostic support tool for LN-classification in HNSCC. However, adequately powered, prospective, and randomized control trials are urgently required to further assess AI's role in LN-classification in locally-advanced HNSCC.

A multimodal model fusing multiphase contrast-enhanced CT and clinical characteristics for predicting lymph node metastases of pancreatic cancer

Objective. To develop a multimodal model that combines multiphase contrast-enhanced computed tomography (CECT) imaging and clinical characteristics, including experts’ experience, to preoperatively predict lymph node metastasis (LNM) in pancreatic cancer patients. Methods. We proposed a new classifier fusion strategy (CFS) based on a new evidential reasoning (ER) rule (CFS-nER) by combining nomogram weights into a previous ER rule-based CFS. Three kernelled support tensor machine-based classifiers with plain, arterial, and venous phases of CECT as the inputs, respectively, were constructed. They were then fused based on the CFS-nER to construct a fusion model of multiphase CECT. The clinical characteristics were analyzed by univariate and multivariable logistic regression to screen risk factors, which were used to construct correspondent risk factor-based classifiers. Finally, the fusion model of the three phases of CECT and each risk factor-based classifier were fused further to construct the multimodal model based on our CFS-nER, named MMM-nER. This study consisted of 186 patients diagnosed with pancreatic cancer from four clinical centers in China, 88 (47.31%) of whom had LNM. Results. The fusion model of the three phases of CECT performed better overall than single and two-phase fusion models; this implies that the three considered phases of CECT were supplementary and complemented one another. The MMM-nER further improved the predictive performance, which implies that our MMM-nER can complement the supplementary information between CECT and clinical characteristics. The MMM-nER had better predictive performance than based on previous classifier fusion strategies, which presents the advantage of our CFS-nER. Conclusion. We proposed a new CFS-nER, based on which the fusion model of the three phases of CECT and MMM-nER were constructed and performed better than all compared methods. MMM-nER achieved an encouraging performance, implying that it can assist clinicians in noninvasively and preoperatively evaluating the lymph node status of pancreatic cancer.

Artificial intelligence and radiomics: fundamentals, applications, and challenges in immunotherapy

Immunotherapy offers the potential for durable clinical benefit but calls into question the association between tumor size and outcome that currently forms the basis for imaging-guided treatment. Artificial intelligence (AI) and radiomics allow for discovery of novel patterns in medical images that can increase radiology’s role in management of patients with cancer, although methodological issues in the literature limit its clinical application. Using keywords related to immunotherapy and radiomics, we performed a literature review of MEDLINE, CENTRAL, and Embase from database inception through February 2022. We removed all duplicates, non-English language reports, abstracts, reviews, editorials, perspectives, case reports, book chapters, and non-relevant studies. From the remaining articles, the following information was extracted: publication information, sample size, primary tumor site, imaging modality, primary and secondary study objectives, data collection strategy (retrospective vs prospective, single center vs multicenter), radiomic signature validation strategy, signature performance, and metrics for calculation of a Radiomics Quality Score (RQS). We identified 351 studies, of which 87 were unique reports relevant to our research question. The median (IQR) of cohort sizes was 101 (57–180). Primary stated goals for radiomics model development were prognostication (n=29, 33.3%), treatment response prediction (n=24, 27.6%), and characterization of tumor phenotype (n=14, 16.1%) or immune environment (n=13, 14.9%). Most studies were retrospective (n=75, 86.2%) and recruited patients from a single center (n=57, 65.5%). For studies with available information on model testing, most (n=54, 65.9%) used a validation set or better. Performance metrics were generally highest for radiomics signatures predicting treatment response or tumor phenotype, as opposed to immune environment and overall prognosis. Out of a possible maximum of 36 points, the median (IQR) of RQS was 12 (10–16). While a rapidly increasing number of promising results offer proof of concept that AI and radiomics could drive precision medicine approaches for a wide range of indications, standardizing the data collection as well as optimizing the methodological quality and rigor are necessary before these results can be translated into clinical practice.

An Investigation on Radiomics Feature Handling for HNSCC Staging Classification

The incidence of Head and Neck Squamous Cell Carcinoma (HNSCC) has been growing in the last few decades. Its diagnosis is usually performed through clinical evaluation and analyzing radiological images, then confirmed by histopathological examination, an invasive and time-consuming operation. The recent advances in the artificial intelligence field are leading to interesting results in the early diagnosis, personalized treatment and monitoring of HNSCC only by analyzing radiological images, without performing a tissue biopsy. The large amount of radiological images and the increasing interest in radiomics approaches can help to develop machine learning (ML) methods to support diagnosis. In this work, we propose an ML method based on the use of radiomics features, extracted from CT and PET images, to classify the disease in terms of pN-Stage, pT-Stage and Overall Stage. After the extraction of radiomics features, a selection step is performed to remove dataset redundancy. Finally, ML methods are employed to complete the classification task. Our pipeline is applied on the “Head-Neck-PET-CT” TCIA open-source dataset, considering a cohort of 201 patients from four different institutions. An AUC of 97%, 83% and 93% in terms of pN-Stage, pT-Stage and Overall Stage classification, respectively, is achieved. The obtained results are promising, showing the potential efficiency of the use of radiomics approaches in staging classification.

Radiomics in Oncological PET Imaging: A Systematic Review—Part 1, Supradiaphragmatic Cancers

Radiomics is an upcoming field in nuclear oncology, both promising and technically challenging. To summarize the already undertaken work on supradiaphragmatic neoplasia and assess its quality, we performed a literature search in the PubMed database up to 18 February 2022. Inclusion criteria were: studies based on human data; at least one specified tumor type; supradiaphragmatic malignancy; performing radiomics on PET imaging. Exclusion criteria were: studies only based on phantom or animal data; technical articles without a clinically oriented question; fewer than 30 patients in the training cohort. A review database containing PMID, year of publication, cancer type, and quality criteria (number of patients, retrospective or prospective nature, independent validation cohort) was constructed. A total of 220 studies met the inclusion criteria. Among them, 119 (54.1%) studies included more than 100 patients, 21 studies (9.5%) were based on prospectively acquired data, and 91 (41.4%) used an independent validation set. Most studies focused on prognostic and treatment response objectives. Because the textural parameters and methods employed are very different from one article to another, it is complicated to aggregate and compare articles. New contributions and radiomics guidelines tend to help improving quality of the reported studies over the years.

Segmentation of metastatic cervical lymph nodes from CT images of oral cancers using deep learning technology

OBJECTIVE

The purpose of this study was to establish a deep learning model for segmenting the cervical lymph nodes of oral cancer patients and diagnosing metastatic or non-metastatic lymph nodes from contrast-enhanced computed tomography (CT) images.

METHODS

CT images of 158 metastatic and 514 non-metastatic lymph nodes were prepared. CT images were assigned to training, validation, and test datasets. The colored images with lymph nodes were prepared together with the original images for the training and validation datasets. Learning was performed for 200 epochs using the neural network U-net. Performance in segmenting lymph nodes and diagnosing metastasis were obtained.

RESULTS

Performance in segmenting metastatic lymph nodes showed recall of 0.742, precision of 0.942, and F1 score of 0.831. The recall of metastatic lymph nodes at level II was 0.875, which was the highest value. The diagnostic performance of identifying metastasis showed an area under the curve (AUC) of 0.950, which was significantly higher than that of radiologists (0.896).

CONCLUSIONS

A deep learning model was created to automatically segment the cervical lymph nodes of oral squamous cell carcinomas. Segmentation performances should still be improved, but the segmented lymph nodes were more accurately diagnosed for metastases compared with evaluation by humans.

Predicting distant metastases in soft-tissue sarcomas from PET-CT scans using constrained hierarchical multi-modality feature learning

Objective.Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of soft-tissue sarcomas (STSs). Distant metastases (DM) are the leading cause of death in STS patients and early detection is important to effectively manage tumors with surgery, radiotherapy and chemotherapy. In this study, we aim to early detect DM in patients with STS using their PET-CT data.Approach.We derive a new convolutional neural network method for early DM detection. The novelty of our method is the introduction of a constrained hierarchical multi-modality feature learning approach to integrate functional imaging (PET) features with anatomical imaging (CT) features. In addition, we removed the reliance on manual input, e.g. tumor delineation, for extracting imaging features.Main results.Our experimental results on a well-established benchmark PET-CT dataset show that our method achieved the highest accuracy (0.896) and AUC (0.903) scores when compared to the state-of-the-art methods (unpaired student's t-testp-value < 0.05).Significance.Our method could be an effective and supportive tool to aid physicians in tumor quantification and in identifying image biomarkers for cancer treatment.

A computed tomography-based radiomic nomogram for predicting lymph node metastasis in patients with early-stage papillary thyroid carcinoma

BACKGROUND

Accurate assessment of lymph node metastasis (LNM) is important for the selection of the optimal therapeutic strategy in patients with papillary thyroid carcinoma (PTC).

PURPOSE

To develop and validate a radiomics nomogram based on computed tomography (CT) for predicting LNM in patients with early-stage PTC.

MATERIAL AND METHODS

A total of 92 patients with pathologically confirmed PTC were divided into a training cohort (n =  64) and validation cohort (n  =  28). Radiomic features of the tumor and peritumoral interstitium were extracted from contrast-enhanced CT images. The radiomic signature was constructed and the radiomic score (Rad-score) was calculated. Combined with the Rad-score and independent clinical factors, a radiomic nomogram was constructed and its performance was assessed by receiver operating characteristic (ROC) curves and calibration plots. The comparison of ROC curves was performed with DeLong's test.

RESULTS

A combined nomogram model of the thyroid tumor and peritumoral interstitium was constructed based on the Rad-score, tumor location, maximum diameter, and T stage, and it had areas under the ROC curve of 0.956 (95% confidence interval [CI] = 0.913-1.000) and 0.876 (95% CI = 0.741-1.000) in the training and validation cohorts, respectively. Decision curve analysis suggested that the combined nomogram model had better clinical usefulness than the other models.

CONCLUSION

A CT-based radiomics nomogram incorporating the radiomic signature and the selected clinical predictors can be a reliable approach to preoperatively predict the LNM status in patients with early-stage PTC, which is helpful for treatment decisions and prognosis.

Clinical Application of Artificial Intelligence in PET Imaging of Head and Neck Cancer

Applications of "artificial intelligence" (AI) have been exponentially expanding in health care. Readily accessible archives of enormous digital data in medical imaging have made radiology a leader in exploring and taking advantage of this technology. AI-assisted radiology has paved the way toward another level of precision in medicine. In this article, the authors aim to review current AI applications in PET imaging of head and neck cancers, beginning with radiomics and followed by deep learning in each section.

Machine Learning Applications for Head and Neck Imaging

The head and neck (HN) consists of a large number of vital anatomic structures within a compact area. Imaging plays a central role in the diagnosis and management of major disorders affecting the HN. This article reviews the recent applications of machine learning (ML) in HN imaging with a focus on deep learning approaches. It categorizes ML applications in HN imaging into deep learning and traditional ML applications and provides examples of each category. It also discusses the main challenges facing the successful deployment of ML-based applications in the clinical setting and provides suggestions for addressing these challenges.

Tumor radiomics signature for artificial neural network-assisted detection of neck metastasis in patient with tongue cancer

BACKGROUND AND PURPOSE

To determine the neck management of tongue cancer, this study attempted to construct an artificial neural network (ANN)-assisted model based on computed tomography (CT) radiomics of primary tumors to predict neck lymph node (LN) status in patients with tongue squamous cell carcinoma (SCC).

MATERIALS AND METHODS

Three hundred thirteen patients with tongue SCC were retrospectively included and randomly divided into training (60%), validation (20%) and internally independent test (20%) sets. In total, 1673 feature values were extracted after the semiautomatic segmentation of primary tumors and set as input layers of a classical 3-layer ANN incorporated with or without clinical LN (cN) status after dimension reduction. The receiver operating characteristic (ROC) curve, accuracy (ACC), sensitivity (SEN), specificity (SPE), area under curve (AUC) and Net Reclassification Index (NRI), were used to evaluate and compare the models.

RESULTS

Four models with different settings were constructed. The ACC, SEN, SPE and AUC reached 84.1%, 93.1%, 76.5% and 0.943 (95% confidence interval: 0.891-0.996, p<.001), respectively, in the test set. The NRI of models compared with radiologists reached 40% (p<.001). The occult nodal metastasis rate was reduced from 30.9% to a minimum of 12.7% in the T1-2 group.

CONCLUSION

ANN-based models that incorporated CT radiomics of primary tumors with traditional LN evaluation were constructed and validated to more precisely predict neck LN metastasis in patients with tongue SCC than with naked eyes, especially in early-stage cancer.

Site-Specific Variation in Radiomic Features of Head and Neck Squamous Cell Carcinoma and Its Impact on Machine Learning Models

Simple Summary

Head and neck squamous cell carcinoma (HNSCC) is the most common mucosal malignancy of the head and neck and a leading cause of cancer death. HNSCC arises from different primary anatomical locations that are typically combined during radiomic analyses assuming that the radiomic features, i.e., quantitative image-based features, are similar based on histopathologic characteristics. However, whether these quantitative features are comparable across tumor sites remains unknown. The aim of our retrospective study was to assess if systematic differences exist between radiomic features based on different tumor sites in HNSCC and how they might affect machine learning model performance in endpoint prediction. Using a population of 605 HNSCC patients, we observed significant differences in radiomic features of tumors from different locations and showed that these differences can impact machine learning model performance. This suggests that tumor site should be considered when developing and evaluating radiomics-based models.

Abstract

Current radiomic studies of head and neck squamous cell carcinomas (HNSCC) are typically based on datasets combining tumors from different locations, assuming that the radiomic features are similar based on histopathologic characteristics. However, molecular pathogenesis and treatment in HNSCC substantially vary across different tumor sites. It is not known if a statistical difference exists between radiomic features from different tumor sites and how they affect machine learning model performance in endpoint prediction. To answer these questions, we extracted radiomic features from contrast-enhanced neck computed tomography scans (CTs) of 605 patients with HNSCC originating from the oral cavity, oropharynx, and hypopharynx/larynx. The difference in radiomic features of tumors from these sites was assessed using statistical analyses and Random Forest classifiers on the radiomic features with 10-fold cross-validation to predict tumor sites, nodal metastasis, and HPV status. We found statistically significant differences (p-value ≤ 0.05) between the radiomic features of HNSCC depending on tumor location. We also observed that differences in quantitative features among HNSCC from different locations impact the performance of machine learning models. This suggests that radiomic features may reveal biologic heterogeneity complementary to current gold standard histopathologic evaluation. We recommend considering tumor site in radiomic studies of HNSCC.

Attention Guided Lymph Node Malignancy Prediction in Head and Neck Cancer

PURPOSE

Accurate lymph node (LN) malignancy classification is essential for treatment target identification in head and neck cancer (HNC) radiation therapy. Given the constraints imposed by relatively small sample sizes in real-world medical applications, to classify LN malignancy status accurately, we proposed an attention-guided classification (AGC) scheme that (1) incorporates human knowledge (ie, LN contours) into model training to guide model's "learning" direction, alleviating the critical requirement of large training samples by deep learning approaches; and (2) does not require accurate delineation of LNs in the inference stage but can highlight the discriminative region nearby the LN, which is important for malignancy determination.

METHODS AND MATERIALS

In the proposed AGC scheme, there is an attention-guided convolutional neural network (agCNN) module, followed by a classification convolutional neural network (cCNN) module. The input of the proposed AGC scheme is a region of interest (ROI) containing the LN and its surrounding tissues. The agCNN is designed to find the discriminative region in the ROI, which outputs an activation map whose voxel values indicate the importance of the voxels in malignancy prediction. Through multiplying the activation map with the ROI, we obtain the input for the cCNN, which finally outputs the LN malignancy probability. To demonstrate the effectiveness of the proposed scheme, we performed experimental studies using positron emission tomography and contrast-enhanced computed tomography from 129 surgical HNC patients, including 791 LNs, with pathologic ground truth of malignancy status. To evaluate the performance, 5-folder cross validation was used.

RESULTS

The sensitivity, specificity, accuracy, and area under the receiver operating characteristic (ROC) curve values obtained by the proposed AGC scheme were 0.91, 0.93, 0.92, and 0.98, respectively, significantly outperforming conventional convolutional neural network and radiomics approaches at a significance level of .05 under a paired ROC comparison statistical test.

CONCLUSIONS

We developed an AGC scheme that can highlight the discriminative region in an image for LN malignancy prediction, outperforming a conventional radiomics method that requires accurate segmentation and a standard convolutional neural network model without involving segmentation.

Applications of artificial intelligence in oncologic 18F-FDG PET/CT imaging: a systematic review

Artificial intelligence (AI) is a growing field of research that is emerging as a promising adjunct to assist physicians in detection and management of patients with cancer. ¹⁸F-FDG PET imaging helps physicians in detection and management of patients with cancer. In this study we discuss the possible applications of AI in ¹⁸F-FDG PET imaging based on the published studies. A systematic literature review was performed in PubMed on early August 2020 to find the relevant studies. A total of 65 studies were available for review against the inclusion criteria which included studies that developed an AI model based on 18F-FDG PET data in cancer to diagnose, differentiate, delineate, stage, assess response to therapy, determine prognosis, or improve image quality. Thirty-two studies met the inclusion criteria and are discussed in this review. The majority of studies are related to lung cancer. Other studied cancers included breast cancer, cervical cancer, head and neck cancer, lymphoma, pancreatic cancer, and sarcoma. All studies were based on human patients except for one which was performed on rats. According to the included studies, machine learning (ML) models can help in detection, differentiation from benign lesions, segmentation, staging, response assessment, and prognosis determination. Despite the potential benefits of AI in cancer imaging and management, the routine implementation of AI-based models and ¹⁸F-FDG PET-derived radiomics in clinical practice is limited at least partially due to lack of standardized, reproducible, generalizable, and precise techniques.

Artificial intelligence in molecular imaging

AI has, to varying degrees, affected all aspects of molecular imaging, from image acquisition to diagnosis. During the last decade, the advent of deep learning in particular has transformed medical image analysis. Although the majority of recent advances have resulted from neural-network models applied to image segmentation, a broad range of techniques has shown promise for image reconstruction, image synthesis, differential-diagnosis generation, and treatment guidance. Applications of AI for drug design indicate the way forward for using AI to facilitate molecular-probe design, which is still in its early stages. Deep-learning models have demonstrated increased efficiency and image quality for PET reconstruction from sinogram data. Generative adversarial networks (GANs), which are paired neural networks that are jointly trained to generate and classify images, have found applications in modality transformation, artifact reduction, and synthetic-PET-image generation. Some AI applications, based either partly or completely on neural-network approaches, have demonstrated superior differential-diagnosis generation relative to radiologists. However, AI models have a history of brittleness, and physicians and patients may not trust AI applications that cannot explain their reasoning. To date, the majority of molecular-imaging applications of AI have been confined to research projects, and are only beginning to find their ways into routine clinical workflows via commercialization and, in some cases, integration into scanner hardware. Evaluation of actual clinical products will yield more realistic assessments of AI’s utility in molecular imaging.

Automated detection and segmentation of thoracic lymph nodes from CT using 3D foveal fully convolutional neural networks

BACKGROUND

In oncology, the correct determination of nodal metastatic disease is essential for patient management, as patient treatment and prognosis are closely linked to the stage of the disease. The aim of the study was to develop a tool for automatic 3D detection and segmentation of lymph nodes (LNs) in computed tomography (CT) scans of the thorax using a fully convolutional neural network based on 3D foveal patches.

METHODS

The training dataset was collected from the Computed Tomography Lymph Nodes Collection of the Cancer Imaging Archive, containing 89 contrast-enhanced CT scans of the thorax. A total number of 4275 LNs was segmented semi-automatically by a radiologist, assessing the entire 3D volume of the LNs. Using this data, a fully convolutional neuronal network based on 3D foveal patches was trained with fourfold cross-validation. Testing was performed on an unseen dataset containing 15 contrast-enhanced CT scans of patients who were referred upon suspicion or for staging of bronchial carcinoma.

RESULTS

The algorithm achieved a good overall performance with a total detection rate of 76.9% for enlarged LNs during fourfold cross-validation in the training dataset with 10.3 false-positives per volume and of 69.9% in the unseen testing dataset. In the training dataset a better detection rate was observed for enlarged LNs compared to smaller LNs, the detection rate for LNs with a short-axis diameter (SAD) ≥ 20 mm and SAD 5-10 mm being 91.6% and 62.2% (p < 0.001), respectively. Best detection rates were obtained for LNs located in Level 4R (83.6%) and Level 7 (80.4%).

CONCLUSIONS

The proposed 3D deep learning approach achieves an overall good performance in the automatic detection and segmentation of thoracic LNs and shows reasonable generalizability, yielding the potential to facilitate detection during routine clinical work and to enable radiomics research without observer-bias.

Computer-aided diagnosis of masses in breast computed tomography imaging: deep learning model with combined handcrafted and convolutional radiomic features

Purpose: A computer-aided diagnosis (CADx) system for breast masses is proposed, which incorporates both handcrafted and convolutional radiomic features embedded into a single deep learning model. Approach: The model combines handcrafted and convolutional radiomic signatures into a multi-view architecture, which retrieves three-dimensional (3D) image information by simultaneously processing multiple two-dimensional mass patches extracted along different planes through the 3D mass volume. Each patch is processed by a stream composed of two concatenated parallel branches: a multi-layer perceptron fed with automatically extracted handcrafted radiomic features, and a convolutional neural network, for which discriminant features are learned from the input patches. All streams are then concatenated together into a final architecture, where all network weights are shared and the learning occurs simultaneously for each stream and branch. The CADx system was developed and tested for diagnosis of breast masses ( N = 284 ) using image datasets acquired with independent dedicated breast computed tomography systems from two different institutions. The diagnostic classification performance of the CADx system was compared against other machine and deep learning architectures adopting handcrafted and convolutional approaches, and three board-certified breast radiologists. Results: On a test set of 82 masses (45 benign, 37 malignant), the proposed CADx system performed better than all other model architectures evaluated, with an increase in the area under the receiver operating characteristics curve (AUC) of 0.05 ± 0.02 , and achieving a final AUC of 0.947, outperforming the three radiologists ( AUC = 0.814 - 0.902 ). Conclusions: In conclusion, the system demonstrated its potential usefulness in breast cancer diagnosis by improving mass malignancy assessment.

Use of deep learning for detection, characterisation and prediction of metastatic disease from computerised tomography: a systematic review

CT is widely used for diagnosis, staging and management of cancer. The presence of metastasis has significant implications on treatment and prognosis. Deep learning (DL), a form of machine learning, where layers of programmed algorithms interpret and recognise patterns, may have a potential role in CT image analysis. This review aims to provide an overview on the use of DL in CT image analysis in the diagnostic evaluation of metastatic disease. A total of 29 studies were included which could be grouped together into three areas of research: the use of deep learning on the detection of metastatic disease from CT imaging, characterisation of lesions on CT into metastasis and prediction of the presence or development of metastasis based on the primary tumour. In conclusion, DL in CT image analysis could have a potential role in evaluating metastatic disease; however, prospective clinical trials investigating its clinical value are required.

A Systematic Review of PET Textural Analysis and Radiomics in Cancer

Background: Although many works have supported the utility of PET radiomics, several authors have raised concerns over the robustness and replicability of the results. This study aimed to perform a systematic review on the topic of PET radiomics and the used methodologies. Methods: PubMed was searched up to 15 October 2020. Original research articles based on human data specifying at least one tumor type and PET image were included, excluding those that apply only first-order statistics and those including fewer than 20 patients. Each publication, cancer type, objective and several methodological parameters (number of patients and features, validation approach, among other things) were extracted. Results: A total of 290 studies were included. Lung (28%) and head and neck (24%) were the most studied cancers. The most common objective was prognosis/treatment response (46%), followed by diagnosis/staging (21%), tumor characterization (18%) and technical evaluations (15%). The average number of patients included was 114 (median = 71; range 20–1419), and the average number of high-order features calculated per study was 31 (median = 26, range 1–286). Conclusions: PET radiomics is a promising field, but the number of patients in most publications is insufficient, and very few papers perform in-depth validations. The role of standardization initiatives will be crucial in the upcoming years.

Deep Learning in Head and Neck Tumor Multiomics Diagnosis and Analysis: Review of the Literature

Head and neck tumors are the sixth most common neoplasms. Multiomics integrates multiple dimensions of clinical, pathologic, radiological, and biological data and has the potential for tumor diagnosis and analysis. Deep learning (DL), a type of artificial intelligence (AI), is applied in medical image analysis. Among the DL techniques, the convolution neural network (CNN) is used for image segmentation, detection, and classification and in computer-aided diagnosis. Here, we reviewed multiomics image analysis of head and neck tumors using CNN and other DL neural networks. We also evaluated its application in early tumor detection, classification, prognosis/metastasis prediction, and the signing out of the reports. Finally, we highlighted the challenges and potential of these techniques.

Predicting lymph node metastasis in patients with oropharyngeal cancer by using a convolutional neural network with associated epistemic and aleatoric uncertainty

There can be significant uncertainty when identifying cervical lymph node (LN) metastases in patients with oropharyngeal squamous cell carcinoma (OPSCC) despite the use of modern imaging modalities such as positron emission tomography (PET) and computed tomography (CT) scans. Grossly involved LNs are readily identifiable during routine imaging, but smaller and less PET-avid LNs are harder to classify. We trained a convolutional neural network (CNN) to detect malignant LNs in patients with OPSCC and used quantitative measures of uncertainty to identify the most reliable predictions. Our dataset consisted of images of 791 LNs from 129 patients with OPSCC who had preoperative PET/CT imaging and detailed pathological reports after neck dissections. These LNs were segmented on PET/CT imaging and then labeled according to the pathology reports. An AlexNet-like CNN was trained to classify LNs as malignant or benign. We estimated epistemic and aleatoric uncertainty by using dropout variational inference and test-time augmentation, respectively. CNN performance was stratified according to the median epistemic and aleatoric uncertainty values calculated using the validation cohort. Our model achieved an area under the receiver operating characteristic (ROC) curve (AUC) of 0.99 on the testing dataset. Sensitivity and specificity were 0.94 and 0.90, respectively. Epistemic and aleatoric uncertainty values were statistically larger for false negative and false positive predictions than for true negative and true positive predictions (p < 0.001). Model sensitivity and specificity were 1.0 and 0.98, respectively, for cases with epistemic uncertainty lower than the median value of the incorrect predictions in the validation dataset. For cases with higher epistemic uncertainty, sensitivity and specificity were 0.67 and 0.41, respectively. Model sensitivity and specificity were 1.0 and 0.98, respectively, for cases with aleatoric uncertainty lower than the median value of the incorrect predictions in the validation dataset. For cases with higher aleatoric uncertainty, sensitivity and specificity were 0.67 and 0.37, respectively. We used a CNN to predict the malignant status of LNs in patients with OPSCC with high accuracy, and we showed that uncertainty can be used to quantify a prediction's reliability. Assigning measures of uncertainty to predictions could improve the accuracy of LN classification by efficiently identifying instances where expert evaluation is needed to corroborate a model's prediction.

3D Deep Learning on Medical Images: A Review

The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for the analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, we provide a brief mathematical description of 3D CNN and provide the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models in general) and possible future trends in the field.

Radiomics at a Glance: A Few Lessons Learned from Learning Approaches

Simple Summary

Radiomics has become a prominent component of medical imaging research and many studies show its specific value as a support tool for clinical decision-making processes. Radiomic data are typically analyzed with statistical and machine learning methods, which change depending on the disease context and the imaging modality. We found a certain bias in the literature towards the use of such methods and believe that this limitation may influence the capacity of producing accurate and reliable decisions. Therefore, in view of the relevance of various types of learning methods, we report their significance and discuss their unrevealed potential.

Abstract

Processing and modeling medical images have traditionally represented complex tasks requiring multidisciplinary collaboration. The advent of radiomics has assigned a central role to quantitative data analytics targeting medical image features algorithmically extracted from large volumes of images. Apart from the ultimate goal of supporting diagnostic, prognostic, and therapeutic decisions, radiomics is computationally attractive due to specific strengths: scalability, efficiency, and precision. Optimization is achieved by highly sophisticated statistical and machine learning algorithms, but it is especially deep learning that stands out as the leading inference approach. Various types of hybrid learning can be considered when building complex integrative approaches aimed to deliver gains in accuracy for both classification and prediction tasks. This perspective reviews some selected learning methods by focusing on both their significance for radiomics and their unveiled potential.

A Literature Review of the Potential Diagnostic Biomarkers of Head and Neck Neoplasms

Head and neck neoplasms have a poor prognosis because of their late diagnosis. Finding a biomarker to detect these tumors in an early phase could improve the prognosis and survival rate. This literature review provides an overview of biomarkers, covering the different -omics fields to diagnose head and neck neoplasms in the early phase. To date, not a single biomarker, nor a panel of biomarkers for the detection of head and neck tumors has been detected with clinical applicability. Limitations for the clinical implementation of the investigated biomarkers are mainly the heterogeneity of the study groups (e.g., small population in which the biomarker was tested, and/or only including high-risk populations) and a low sensitivity and/or specificity of the biomarkers under study. Further research on biomarkers to diagnose head and neck neoplasms in an early stage, is therefore needed.