Tumor Segmentation and Feature Extraction from Whole-Body FDG-PET/CT Using Cascaded 2D and 3D Convolutional Neural Networks

How Does Target Lesion Selection Affect RECIST? A Computer Simulation Study

OBJECTIVES

Response Evaluation Criteria in Solid Tumors (RECIST) is grounded on the assumption that target lesion selection is objective and representative of the change in total tumor burden (TTB) during therapy. A computer simulation model was designed to challenge this assumption, focusing on a particular aspect of subjectivity: target lesion selection.

MATERIALS AND METHODS

Disagreement among readers and the disagreement between individual reader measurements and TTB were analyzed as a function of the total number of lesions, affected organs, and lesion growth.

RESULTS

Disagreement rises when the number of lesions increases, when lesions are concentrated on a few organs, and when lesion growth borders the thresholds of progressive disease and partial response. There is an intrinsic methodological error in the estimation of TTB via RECIST 1.1, which depends on the number of lesions and their distributions. For example, for a fixed number of lesions at 5 and 15, distributed over a maximum of 4 organs, the error rates are observed to be 7.8% and 17.3%, respectively.

CONCLUSIONS

Our results demonstrate that RECIST can deliver an accurate estimate of TTB in localized disease, but fails in cases of distal metastases and multiple organ involvement. This is worsened by the "selection of the largest lesions," which introduces a bias that makes it hardly possible to perform an accurate estimate of the TTB. Including more (if not all) lesions in the quantitative analysis of tumor burden is desirable.

TMTV-Net: fully automated total metabolic tumor volume segmentation in lymphoma PET/CT images - a multi-center generalizability analysis

PURPOSE

Total metabolic tumor volume (TMTV) segmentation has significant value enabling quantitative imaging biomarkers for lymphoma management. In this work, we tackle the challenging task of automated tumor delineation in lymphoma from PET/CT scans using a cascaded approach.

METHODS

Our study included 1418 2-[18F]FDG PET/CT scans from four different centers. The dataset was divided into 900 scans for development/validation/testing phases and 518 for multi-center external testing. The former consisted of 450 lymphoma, lung cancer, and melanoma scans, along with 450 negative scans, while the latter consisted of lymphoma patients from different centers with diffuse large B cell, primary mediastinal large B cell, and classic Hodgkin lymphoma cases. Our approach involves resampling PET/CT images into different voxel sizes in the first step, followed by training multi-resolution 3D U-Nets on each resampled dataset using a fivefold cross-validation scheme. The models trained on different data splits were ensemble. After applying soft voting to the predicted masks, in the second step, we input the probability-averaged predictions, along with the input imaging data, into another 3D U-Net. Models were trained with semi-supervised loss. We additionally considered the effectiveness of using test time augmentation (TTA) to improve the segmentation performance after training. In addition to quantitative analysis including Dice score (DSC) and TMTV comparisons, the qualitative evaluation was also conducted by nuclear medicine physicians.

RESULTS

Our cascaded soft-voting guided approach resulted in performance with an average DSC of 0.68 ± 0.12 for the internal test data from developmental dataset, and an average DSC of 0.66 ± 0.18 on the multi-site external data (n = 518), significantly outperforming (p < 0.001) state-of-the-art (SOTA) approaches including nnU-Net and SWIN UNETR. While TTA yielded enhanced performance gains for some of the comparator methods, its impact on our cascaded approach was found to be negligible (DSC: 0.66 ± 0.16). Our approach reliably quantified TMTV, with a correlation of 0.89 with the ground truth (p < 0.001). Furthermore, in terms of visual assessment, concordance between quantitative evaluations and clinician feedback was observed in the majority of cases. The average relative error (ARE) and the absolute error (AE) in TMTV prediction on external multi-centric dataset were ARE = 0.43 ± 0.54 and AE = 157.32 ± 378.12 (mL) for all the external test data (n = 518), and ARE = 0.30 ± 0.22 and AE = 82.05 ± 99.78 (mL) when the 10% outliers (n = 53) were excluded.

CONCLUSION

TMTV-Net demonstrates strong performance and generalizability in TMTV segmentation across multi-site external datasets, encompassing various lymphoma subtypes. A negligible reduction of 2% in overall performance during testing on external data highlights robust model generalizability across different centers and cancer types, likely attributable to its training with resampled inputs. Our model is publicly available, allowing easy multi-site evaluation and generalizability analysis on datasets from different institutions.

Deep Semisupervised Transfer Learning for Fully Automated Whole-Body Tumor Quantification and Prognosis of Cancer on PET/CT

Automatic detection and characterization of cancer are important clinical needs to optimize early treatment. We developed a deep, semisupervised transfer learning approach for fully automated, whole-body tumor segmentation and prognosis on PET/CT. Methods: This retrospective study consisted of 611 18F-FDG PET/CT scans of patients with lung cancer, melanoma, lymphoma, head and neck cancer, and breast cancer and 408 prostate-specific membrane antigen (PSMA) PET/CT scans of patients with prostate cancer. The approach had a nnU-net backbone and learned the segmentation task on 18F-FDG and PSMA PET/CT images using limited annotations and radiomics analysis. True-positive rate and Dice similarity coefficient were assessed to evaluate segmentation performance. Prognostic models were developed using imaging measures extracted from predicted segmentations to perform risk stratification of prostate cancer based on follow-up prostate-specific antigen levels, survival estimation of head and neck cancer by the Kaplan-Meier method and Cox regression analysis, and pathologic complete response prediction of breast cancer after neoadjuvant chemotherapy. Overall accuracy and area under the receiver-operating-characteristic (AUC) curve were assessed. Results: Our approach yielded median true-positive rates of 0.75, 0.85, 0.87, and 0.75 and median Dice similarity coefficients of 0.81, 0.76, 0.83, and 0.73 for patients with lung cancer, melanoma, lymphoma, and prostate cancer, respectively, on the tumor segmentation task. The risk model for prostate cancer yielded an overall accuracy of 0.83 and an AUC of 0.86. Patients classified as low- to intermediate- and high-risk had mean follow-up prostate-specific antigen levels of 18.61 and 727.46 ng/mL, respectively (P < 0.05). The risk score for head and neck cancer was significantly associated with overall survival by univariable and multivariable Cox regression analyses (P < 0.05). Predictive models for breast cancer predicted pathologic complete response using only pretherapy imaging measures and both pre- and posttherapy measures with accuracies of 0.72 and 0.84 and AUCs of 0.72 and 0.76, respectively. Conclusion: The proposed approach demonstrated accurate tumor segmentation and prognosis in patients across 6 cancer types on 18F-FDG and PSMA PET/CT scans.

Multicenter development of a PET-based risk assessment tool for product-specific outcome prediction in large B-cell lymphoma patients undergoing CAR T-cell therapy

PURPOSE

The emergence of chimeric antigen receptor (CAR) T-cell therapy fundamentally changed the management of individuals with relapsed and refractory large B-cell lymphoma (LBCL). However, real-world data have shown divergent outcomes for the approved products. The present study therefore set out to evaluate potential risk factors in a larger cohort.

METHODS

Our analysis set included 88 patients, treated in four German university hospitals and one Italian center, who had undergone 2-[18F]fluoro-2-deoxy-D-glucose positron emission tomography (PET) before CAR T-cell therapy with tisagenlecleucel or axicabtagene ciloleucel. We first determined the predictive value of conventional risk factors, treatment lines, and response to bridging therapy for progression-free survival (PFS) through forward selection based on Cox regression. In a second step, the additive potential of two common PET parameters was assessed. Their optimal dichotomizing thresholds were calculated individually for each CAR T-cell product.

RESULTS

Extra-nodal involvement emerged as the most relevant of the conventional tumor and patient characteristics. Moreover, we found that inclusion of metabolic tumor volume (MTV) further improves outcome prediction. The hazard ratio for a PFS event was 1.68 per unit increase of our proposed risk score (95% confidence interval [1.20, 2.35], P = 0.003), which comprised both extra-nodal disease and lymphoma burden. While the most suitable MTV cut-off among patients receiving tisagenlecleucel was 11 mL, a markedly higher threshold of 259 mL showed optimal predictive performance in those undergoing axicabtagene ciloleucel treatment.

CONCLUSION

Our analysis demonstrates that the presence of more than one extra-nodal lesion and higher MTV in LBCL are associated with inferior outcome after CAR T-cell treatment. Based on an assessment tool including these two factors, patients can be assigned to one of three risk groups. Importantly, as shown by our study, metabolic tumor burden might facilitate CAR T-cell product selection and reflect the individual need for bridging therapy.

Transfer learning-based PET/CT three-dimensional convolutional neural network fusion of image and clinical information for prediction of EGFR mutation in lung adenocarcinoma

BACKGROUND

To introduce a three-dimensional convolutional neural network (3D CNN) leveraging transfer learning for fusing PET/CT images and clinical data to predict EGFR mutation status in lung adenocarcinoma (LADC).

METHODS

Retrospective data from 516 LADC patients, encompassing preoperative PET/CT images, clinical information, and EGFR mutation status, were divided into training (n = 404) and test sets (n = 112). Several deep learning models were developed utilizing transfer learning, involving CT-only and PET-only models. A dual-stream model fusing PET and CT and a three-stream transfer learning model (TS_TL) integrating clinical data were also developed. Image preprocessing includes semi-automatic segmentation, resampling, and image cropping. Considering the impact of class imbalance, the performance of the model was evaluated using ROC curves and AUC values.

RESULTS

TS_TL model demonstrated promising performance in predicting the EGFR mutation status, with an AUC of 0.883 (95%CI = 0.849-0.917) in the training set and 0.730 (95%CI = 0.629-0.830) in the independent test set. Particularly in advanced LADC, the model achieved an AUC of 0.871 (95%CI = 0.823-0.919) in the training set and 0.760 (95%CI = 0.638-0.881) in the test set. The model identified distinct activation areas in solid or subsolid lesions associated with wild and mutant types. Additionally, the patterns captured by the model were significantly altered by effective tyrosine kinase inhibitors treatment, leading to notable changes in predicted mutation probabilities.

CONCLUSION

PET/CT deep learning model can act as a tool for predicting EGFR mutation in LADC. Additionally, it offers clinicians insights for treatment decisions through evaluations both before and after treatment.

Improved automated tumor segmentation in whole-body 3D scans using multi-directional 2D projection-based priors

Early cancer detection, guided by whole-body imaging, is important for the overall survival and well-being of the patients. While various computer-assisted systems have been developed to expedite and enhance cancer diagnostics and longitudinal monitoring, the detection and segmentation of tumors, especially from whole-body scans, remain challenging. To address this, we propose a novel end-to-end automated framework that first generates a tumor probability distribution map (TPDM), incorporating prior information about the tumor characteristics (e.g. size, shape, location). Subsequently, the TPDM is integrated with a state-of-the-art 3D segmentation network along with the original PET/CT or PET/MR images. This aims to produce more meaningful tumor segmentation masks compared to using the baseline 3D segmentation network alone. The proposed method was evaluated on three independent cohorts (autoPET, CAR-T, cHL) of images containing different cancer forms, obtained with different imaging modalities, and acquisition parameters and lesions annotated by different experts. The evaluation demonstrated the superiority of our proposed method over the baseline model by significant margins in terms of Dice coefficient, and lesion-wise sensitivity and precision. Many of the extremely small tumor lesions (i.e. the most difficult to segment) were missed by the baseline model but detected by the proposed model without additional false positives, resulting in clinically more relevant assessments. On average, an improvement of 0.0251 (autoPET), 0.144 (CAR-T), and 0.0528 (cHL) in overall Dice was observed. In conclusion, the proposed TPDM-based approach can be integrated with any state-of-the-art 3D UNET with potentially more accurate and robust segmentation results.

Deep learning for [18F]fluorodeoxyglucose-PET-CT classification in patients with lymphoma: a dual-centre retrospective analysis

BACKGROUND

The rising global cancer burden has led to an increasing demand for imaging tests such as [18F]fluorodeoxyglucose ([18F]FDG)-PET-CT. To aid imaging specialists in dealing with high scan volumes, we aimed to train a deep learning artificial intelligence algorithm to classify [18F]FDG-PET-CT scans of patients with lymphoma with or without hypermetabolic tumour sites.

METHODS

In this retrospective analysis we collected 16 583 [18F]FDG-PET-CTs of 5072 patients with lymphoma who had undergone PET-CT before or after treatment at the Memorial Sloa Kettering Cancer Center, New York, NY, USA. Using maximum intensity projection (MIP), three dimensional (3D) PET, and 3D CT data, our ResNet34-based deep learning model (Lymphoma Artificial Reader System [LARS]) for [18F]FDG-PET-CT binary classification (Deauville 1-3 vs 4-5), was trained on 80% of the dataset, and tested on 20% of this dataset. For external testing, 1000 [18F]FDG-PET-CTs were obtained from a second centre (Medical University of Vienna, Vienna, Austria). Seven model variants were evaluated, including MIP-based LARS-avg (optimised for accuracy) and LARS-max (optimised for sensitivity), and 3D PET-CT-based LARS-ptct. Following expert curation, areas under the curve (AUCs), accuracies, sensitivities, and specificities were calculated.

FINDINGS

In the internal test cohort (3325 PET-CTs, 1012 patients), LARS-avg achieved an AUC of 0·949 (95% CI 0·942-0·956), accuracy of 0·890 (0·879-0·901), sensitivity of 0·868 (0·851-0·885), and specificity of 0·913 (0·899-0·925); LARS-max achieved an AUC of 0·949 (0·942-0·956), accuracy of 0·868 (0·858-0·879), sensitivity of 0·909 (0·896-0·924), and specificity of 0·826 (0·808-0·843); and LARS-ptct achieved an AUC of 0·939 (0·930-0·948), accuracy of 0·875 (0·864-0·887), sensitivity of 0·836 (0·817-0·855), and specificity of 0·915 (0·901-0·927). In the external test cohort (1000 PET-CTs, 503 patients), LARS-avg achieved an AUC of 0·953 (0·938-0·966), accuracy of 0·907 (0·888-0·925), sensitivity of 0·874 (0·843-0·904), and specificity of 0·949 (0·921-0·960); LARS-max achieved an AUC of 0·952 (0·937-0·965), accuracy of 0·898 (0·878-0·916), sensitivity of 0·899 (0·871-0·926), and specificity of 0·897 (0·871-0·922); and LARS-ptct achieved an AUC of 0·932 (0·915-0·948), accuracy of 0·870 (0·850-0·891), sensitivity of 0·827 (0·793-0·863), and specificity of 0·913 (0·889-0·937).

INTERPRETATION

Deep learning accurately distinguishes between [18F]FDG-PET-CT scans of lymphoma patients with and without hypermetabolic tumour sites. Deep learning might therefore be potentially useful to rule out the presence of metabolically active disease in such patients, or serve as a second reader or decision support tool.

FUNDING

National Institutes of Health-National Cancer Institute Cancer Center Support Grant.

Tumor Location Relative to the Spleen Is a Prognostic Factor in Lymphoma Patients: A Demonstration from the REMARC Trial

Baseline [18F]FDG PET/CT radiomic features can improve the survival prediction in patients with diffuse large B-cell lymphoma (DLBCL). The purpose of this study was to investigate whether characterizing tumor locations relative to the spleen location in baseline [18F]FDG PET/CT images predicts survival in patients with DLBCL and improves the predictive value of total metabolic tumor volume (TMTV) and age-adjusted international prognostic index (IPI). Methods: This retrospective study included 301 DLBCL patients from the REMARC (NCT01122472) cohort. Physicians delineated the tumor regions, whereas the spleen was automatically segmented using an open-access artificial intelligence algorithm. We systematically measured the distance between the centroid of the spleen and all other lesions, defining the SD of these distances as the lesion spread (SpreadSpleen). We calculated the maximum distance between the spleen and another lesion (Dspleen) for each patient and normalized it with the body surface area, resulting in standardized Dspleen (sDspleen). The predictive value of each PET/CT feature for progression-free survival (PFS) and overall survival (OS) was evaluated through univariate and multivariate time-dependent Cox models and Kaplan-Meier analysis. Results: In total, 282 patients (mean age, 68.33 ± 5.41 y; 164 men) were evaluated. The artificial intelligence algorithm successfully segmented the spleen in 96% of the patients. SpreadSpleen, Dspleen, and sDspleen were correlated neither with TMTV (Pearson ρ < 0.23) nor with IPI (Pearson ρ < 0.15). When median values were used as the cutoff, SpreadSpleen, Dspleen, and sDspleen all significantly classified patients into 2 risk groups for PFS and OS (P < 0.001). They complemented TMTV and IPI to classify the patients into 3 risk groups for PFS and OS (P < 0.001). Integrating SpreadSpleen, Dspleen, or sDspleen into a Cox model on the basis of TMTV, IPI, and TMTV combined with IPI significantly improved the concordance index for PFS and OS (P < 0.05). Conclusion: Baseline PET/CT features that characterize tumor spread and dissemination relative to the spleen strongly predicted survival in patients with DLBCL. Integrating these features with TMTV and IPI further improved survival prediction.

Multi-modal tumor segmentation methods based on deep learning: a narrative review

Background and Objective

Automatic tumor segmentation is a critical component in clinical diagnosis and treatment. Although single-modal imaging provides useful information, multi-modal imaging provides a more comprehensive understanding of the tumor. Multi-modal tumor segmentation has been an essential topic in medical image processing. With the remarkable performance of deep learning (DL) methods in medical image analysis, multi-modal tumor segmentation based on DL has attracted significant attention. This study aimed to provide an overview of recent DL-based multi-modal tumor segmentation methods.

Methods

In in the PubMed and Google Scholar databases, the keywords "multi-modal", "deep learning", and "tumor segmentation" were used to systematically search English articles in the past 5 years. The date range was from 1 January 2018 to 1 June 2023. A total of 78 English articles were reviewed.

Key Content and Findings

We introduce public datasets, evaluation methods, and multi-modal data processing. We also summarize common DL network structures, techniques, and multi-modal image fusion methods used in different tumor segmentation tasks. Finally, we conclude this study by presenting perspectives for future research.

Conclusions

In multi-modal tumor segmentation tasks, DL technique is a powerful method. With the fusion methods of different modal data, the DL framework can effectively use the characteristics of different modal data to improve the accuracy of tumor segmentation.

Hyper-Dense_Lung_Seg: Multimodal-Fusion-Based Modified U-Net for Lung Tumour Segmentation Using Multimodality of CT-PET Scans

The majority of cancer-related deaths globally are due to lung cancer, which also has the second-highest mortality rate. The segmentation of lung tumours, treatment evaluation, and tumour stage classification have become significantly more accessible with the advent of PET/CT scans. With the advent of PET/CT scans, it is possible to obtain both functioning and anatomic data during a single examination. However, integrating images from different modalities can indeed be time-consuming for medical professionals and remains a challenging task. This challenge arises from several factors, including differences in image acquisition techniques, image resolutions, and the inherent variations in the spectral and temporal data captured by different imaging modalities. Artificial Intelligence (AI) methodologies have shown potential in the automation of image integration and segmentation. To address these challenges, multimodal fusion approach-based U-Net architecture (early fusion, late fusion, dense fusion, hyper-dense fusion, and hyper-dense VGG16 U-Net) are proposed for lung tumour segmentation. Dice scores of 73% show that hyper-dense VGG16 U-Net is superior to the other four proposed models. The proposed method can potentially aid medical professionals in detecting lung cancer at an early stage.

Automatic Detection of Pancreatic Lesions and Main Pancreatic Duct Dilatation on Portal Venous CT Scans Using Deep Learning

OBJECTIVES

This study proposes and evaluates a deep learning method to detect pancreatic neoplasms and to identify main pancreatic duct (MPD) dilatation on portal venous computed tomography scans.

MATERIALS AND METHODS

A total of 2890 portal venous computed tomography scans from 9 institutions were acquired, among which 2185 had a pancreatic neoplasm and 705 were healthy controls. Each scan was reviewed by one in a group of 9 radiologists. Physicians contoured the pancreas, pancreatic lesions if present, and the MPD if visible. They also assessed tumor type and MPD dilatation. Data were split into a training and independent testing set of 2134 and 756 cases, respectively.A method to detect pancreatic lesions and MPD dilatation was built in 3 steps. First, a segmentation network was trained in a 5-fold cross-validation manner. Second, outputs of this network were postprocessed to extract imaging features: a normalized lesion risk, the predicted lesion diameter, and the MPD diameter in the head, body, and tail of the pancreas. Third, 2 logistic regression models were calibrated to predict lesion presence and MPD dilatation, respectively. Performance was assessed on the independent test cohort using receiver operating characteristic analysis. The method was also evaluated on subgroups defined based on lesion types and characteristics.

RESULTS

The area under the curve of the model detecting lesion presence in a patient was 0.98 (95% confidence interval [CI], 0.97-0.99). A sensitivity of 0.94 (469 of 493; 95% CI, 0.92-0.97) was reported. Similar values were obtained in patients with small (less than 2 cm) and isodense lesions with a sensitivity of 0.94 (115 of 123; 95% CI, 0.87-0.98) and 0.95 (53 of 56, 95% CI, 0.87-1.0), respectively. The model sensitivity was also comparable across lesion types with values of 0.94 (95% CI, 0.91-0.97), 1.0 (95% CI, 0.98-1.0), 0.96 (95% CI, 0.97-1.0) for pancreatic ductal adenocarcinoma, neuroendocrine tumor, and intraductal papillary neoplasm, respectively. Regarding MPD dilatation detection, the model had an area under the curve of 0.97 (95% CI, 0.96-0.98).

CONCLUSIONS

The proposed approach showed high quantitative performance to identify patients with pancreatic neoplasms and to detect MPD dilatation on an independent test cohort. Performance was robust across subgroups of patients with different lesion characteristics and types. Results confirmed the interest to combine a direct lesion detection approach with secondary features such as the MPD diameter, thus indicating a promising avenue for the detection of pancreatic cancer at early stages.

Machine Learning CT-Based Automatic Nodal Segmentation and PET Semi-Quantification of Intraoperative 68Ga-PSMA-11 PET/CT Images in High-Risk Prostate Cancer: A Pilot Study

High-resolution intraoperative PET/CT specimen imaging, coupled with prostate-specific membrane antigen (PSMA) molecular targeting, holds great potential for the rapid ex vivo identification of disease localizations in high-risk prostate cancer patients undergoing surgery. However, the accurate analysis of radiotracer uptake would require time-consuming manual volumetric segmentation of 3D images. The aim of this study was to test the feasibility of using machine learning to perform automatic nodal segmentation of intraoperative 68Ga-PSMA-11 PET/CT specimen images. Six (n = 6) lymph-nodal specimens were imaged in the operating room after an e.v. injection of 2.1 MBq/kg of 68Ga-PSMA-11. A machine learning-based approach for automatic lymph-nodal segmentation was developed using only open-source Python libraries (Scikit-learn, SciPy, Scikit-image). The implementation of a k-means clustering algorithm (n = 3 clusters) allowed to identify lymph-nodal structures by leveraging differences in tissue density. Refinement of the segmentation masks was performed using morphological operations and 2D/3D-features filtering. Compared to manual segmentation (ITK-SNAP v4.0.1), the automatic segmentation model showed promising results in terms of weighted average precision (97-99%), recall (68-81%), Dice coefficient (80-88%) and Jaccard index (67-79%). Finally, the ML-based segmentation masks allowed to automatically compute semi-quantitative PET metrics (i.e., SUVmax), thus holding promise for facilitating the semi-quantitative analysis of PET/CT images in the operating room.

3D PET/CT tumor segmentation based on nnU-Net with GCN refinement

Objective. Whole-body positron emission tomography/computed tomography (PET/CT) scans are an important tool for diagnosing various malignancies (e.g. malignant melanoma, lymphoma, or lung cancer), and accurate segmentation of tumors is a key part of subsequent treatment. In recent years, convolutional neural network based segmentation methods have been extensively investigated. However, these methods often give inaccurate segmentation results, such as oversegmentation and undersegmentation. To address these issues, we propose a postprocessing method based on a graph convolutional network (GCN) to refine inaccurate segmentation results and improve the overall segmentation accuracy.Approach. First, nnU-Net is used as an initial segmentation framework, and the uncertainty in the segmentation results is analyzed. Certain and uncertain pixels are used to establish the nodes of a graph. Each node and its 6 neighbors form an edge, and 32 nodes are randomly selected as uncertain nodes to form edges. The highly uncertain nodes are used as the subsequent refinement targets. Second, the nnU-Net results of the certain nodes are used as labels to form a semisupervised graph network problem, and the uncertain part is optimized by training the GCN to improve the segmentation performance. This describes our proposed nnU-Net + GCN segmentation framework.Main results.We perform tumor segmentation experiments with the PET/CT dataset from the MICCIA2022 autoPET challenge. Among these data, 30 cases are randomly selected for testing, and the experimental results show that the false-positive rate is effectively reduced with nnU-Net + GCN refinement. In quantitative analysis, there is an improvement of 2.1% for the average Dice score, 6.4 for the 95% Hausdorff distance (HD95), and 1.7 for the average symmetric surface distance.Significance. The quantitative and qualitative evaluation results show that GCN postprocessing methods can effectively improve the tumor segmentation performance.

Quantitative PET-based biomarkers in lymphoma: getting ready for primetime

The use of functional quantitative biomarkers extracted from routine PET-CT scans to characterize clinical responses in patients with lymphoma is gaining increased attention, and these biomarkers can outperform established clinical risk factors. Total metabolic tumour volume enables individualized estimation of survival outcomes in patients with lymphoma and has shown the potential to predict response to therapy suitable for  risk-adapted treatment approaches in clinical trials. The deployment of machine learning tools in molecular imaging research can assist in recognizing complex patterns and, with image classification, in tumour identification and segmentation of data from PET-CT scans. Initial studies using fully automated approaches to calculate metabolic tumour volume and other PET-based biomarkers have demonstrated appropriate correlation with calculations from experts, warranting further testing in large-scale studies. The extraction of computer-based quantitative tumour characterization through radiomics can provide a comprehensive view of phenotypic heterogeneity that better captures the molecular and functional features of the disease. Additionally, radiomics can be integrated with genomic data to provide more accurate prognostic information. Further improvements in PET-based biomarkers are imminent, although their incorporation into clinical decision-making currently has methodological shortcomings that need to be addressed with confirmatory prospective validation in selected patient populations. In this Review, we discuss the current knowledge, challenges and opportunities in the integration of quantitative PET-based biomarkers in clinical trials and the routine management of patients with lymphoma.

Development and Validation of a Predictive Model for Metastatic Melanoma Patients Treated with Pembrolizumab Based on Automated Analysis of Whole-Body [18F]FDG PET/CT Imaging and Clinical Features

BACKGROUND

Antibodies that inhibit the programmed cell death protein 1 (PD-1) receptor offer a significant survival benefit, potentially cure (i.e., durable disease-free survival following treatment discontinuation), a substantial proportion of patients with advanced melanoma. Most patients however fail to respond to such treatment or acquire resistance. Previously, we reported that baseline total metabolic tumour volume (TMTV) determined by whole-body [18F]FDG PET/CT was independently correlated with survival and able to predict the futility of treatment. Manual delineation of [18F]FDG-avid lesions is however labour intensive and not suitable for routine use. A predictive survival model is proposed based on automated analysis of baseline, whole-body [18F]FDG images.

METHODS

Lesions were segmented on [18F]FDG PET/CT using a deep-learning approach and derived features were investigated through Kaplan-Meier survival estimates with univariate logrank test and Cox regression analyses. Selected parameters were evaluated in multivariate Cox survival regressors.

RESULTS

In the development set of 69 patients, overall survival prediction based on TMTV, lactate dehydrogenase levels and presence of brain metastases achieved an area under the curve of 0.78 at one year, 0.70 at two years. No statistically significant difference was observed with respect to using manually segmented lesions. Internal validation on 31 patients yielded scores of 0.76 for one year and 0.74 for two years.

CONCLUSIONS

Automatically extracted TMTV based on whole-body [18F]FDG PET/CT can aid in building predictive models that can support therapeutic decisions in patients treated with immune-checkpoint blockade.

Role of Artificial Intelligence in PET/CT Imaging for Management of Lymphoma

Our review shows that AI-based analysis of lymphoma whole-body FDG-PET/CT can inform all phases of clinical management including staging, prognostication, treatment planning, and treatment response evaluation. We highlight advancements in the role of neural networks for performing automated image segmentation to calculate PET-based imaging biomarkers such as the total metabolic tumor volume (TMTV). AI-based image segmentation methods are at levels where they can be semi-automatically implemented with minimal human inputs and nearing the level of a second-opinion radiologist. Advances in automated segmentation methods are particularly apparent in the discrimination of lymphomatous vs non-lymphomatous FDG-avid regions, which carries through to automated staging. Automated TMTV calculators, in addition to automated calculation of measures such as Dmax are informing robust models of progression-free survival which can then feed into improved treatment planning.

Clinical application of AI-based PET images in oncological patients

Based on the advantages of revealing the functional status and molecular expression of tumor cells, positron emission tomography (PET) imaging has been performed in numerous types of malignant diseases for diagnosis and monitoring. However, insufficient image quality, the lack of a convincing evaluation tool and intra- and interobserver variation in human work are well-known limitations of nuclear medicine imaging and restrict its clinical application. Artificial intelligence (AI) has gained increasing interest in the field of medical imaging due to its powerful information collection and interpretation ability. The combination of AI and PET imaging potentially provides great assistance to physicians managing patients. Radiomics, an important branch of AI applied in medical imaging, can extract hundreds of abstract mathematical features of images for further analysis. In this review, an overview of the applications of AI in PET imaging is provided, focusing on image enhancement, tumor detection, response and prognosis prediction and correlation analyses with pathology or specific gene mutations in several types of tumors. Our aim is to describe recent clinical applications of AI-based PET imaging in malignant diseases and to focus on the description of possible future developments.

The application of machine learning to imaging in hematological oncology: A scoping review

Background

Here, we conducted a scoping review to (i) establish which machine learning (ML) methods have been applied to hematological malignancy imaging; (ii) establish how ML is being applied to hematological cancer radiology; and (iii) identify addressable research gaps.

Methods

The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews guidelines. The inclusion criteria were (i) pediatric and adult patients with suspected or confirmed hematological malignancy undergoing imaging (population); (ii) any study using ML techniques to derive models using radiological images to apply to the clinical management of these patients (concept); and (iii) original research articles conducted in any setting globally (context). Quality Assessment of Diagnostic Accuracy Studies 2 criteria were used to assess diagnostic and segmentation studies, while the Newcastle-Ottawa scale was used to assess the quality of observational studies.

Results

Of 53 eligible studies, 33 applied diverse ML techniques to diagnose hematological malignancies or to differentiate them from other diseases, especially discriminating gliomas from primary central nervous system lymphomas (n=18); 11 applied ML to segmentation tasks, while 9 applied ML to prognostication or predicting therapeutic responses, especially for diffuse large B-cell lymphoma. All studies reported discrimination statistics, but no study calculated calibration statistics. Every diagnostic/segmentation study had a high risk of bias due to their case-control design; many studies failed to provide adequate details of the reference standard; and only a few studies used independent validation.

Conclusion

To deliver validated ML-based models to radiologists managing hematological malignancies, future studies should (i) adhere to standardized, high-quality reporting guidelines such as the Checklist for Artificial Intelligence in Medical Imaging; (ii) validate models in independent cohorts; (ii) standardize volume segmentation methods for segmentation tasks; (iv) establish comprehensive prospective studies that include different tumor grades, comparisons with radiologists, optimal imaging modalities, sequences, and planes; (v) include side-by-side comparisons of different methods; and (vi) include low- and middle-income countries in multicentric studies to enhance generalizability and reduce inequity.

A whole-body FDG-PET/CT Dataset with manually annotated Tumor Lesions

We describe a publicly available dataset of annotated Positron Emission Tomography/Computed Tomography (PET/CT) studies. 1014 whole body Fluorodeoxyglucose (FDG)-PET/CT datasets (501 studies of patients with malignant lymphoma, melanoma and non small cell lung cancer (NSCLC) and 513 studies without PET-positive malignant lesions (negative controls)) acquired between 2014 and 2018 were included. All examinations were acquired on a single, state-of-the-art PET/CT scanner. The imaging protocol consisted of a whole-body FDG-PET acquisition and a corresponding diagnostic CT scan. All FDG-avid lesions identified as malignant based on the clinical PET/CT report were manually segmented on PET images in a slice-per-slice (3D) manner. We provide the anonymized original DICOM files of all studies as well as the corresponding DICOM segmentation masks. In addition, we provide scripts for image processing and conversion to different file formats (NIfTI, mha, hdf5). Primary diagnosis, age and sex are provided as non-imaging information. We demonstrate how this dataset can be used for deep learning-based automated analysis of PET/CT data and provide the trained deep learning model.

Measurement(s)	tumor lesions
Technology Type(s)	PET/CT
Sample Characteristic - Organism	Homo sapiens

Full automation of total metabolic tumor volume from FDG-PET/CT in DLBCL for baseline risk assessments

Background

Current radiological assessments of ¹⁸fluorodeoxyglucose-positron emission tomography (FDG-PET) imaging data in diffuse large B-cell lymphoma (DLBCL) can be time consuming, do not yield real-time information regarding disease burden and organ involvement, and hinder the use of FDG-PET to potentially limit the reliance on invasive procedures (e.g. bone marrow biopsy) for risk assessment.

Methods

Our aim is to enable real-time assessment of imaging-based risk factors at a large scale and we propose a fully automatic artificial intelligence (AI)-based tool to rapidly extract FDG-PET imaging metrics in DLBCL. On availability of a scan, in combination with clinical data, our approach generates clinically informative risk scores with minimal resource requirements. Overall, 1268 patients with previously untreated DLBCL from the phase III GOYA trial (NCT01287741) were included in the analysis (training: n = 846; hold-out: n = 422).

Results

Our AI-based model comprising imaging and clinical variables yielded a tangible prognostic improvement compared to clinical models without imaging metrics. We observed a risk increase for progression-free survival (PFS) with hazard ratios [HR] of 1.87 (95% CI: 1.31–2.67) vs 1.38 (95% CI: 0.98–1.96) (C-index: 0.59 vs 0.55), and a risk increase for overall survival (OS) (HR: 2.16 (95% CI: 1.37–3.40) vs 1.40 (95% CI: 0.90–2.17); C-index: 0.59 vs 0.55). The combined model defined a high-risk population with 35% and 42% increased odds of a 4-year PFS and OS event, respectively, versus the International Prognostic Index components alone. The method also identified a subpopulation with a 2-year Central Nervous System (CNS)-relapse probability of 17.1%.

Conclusion

Our tool enables an enhanced risk stratification compared with IPI, and the results indicate that imaging can be used to improve the prediction of central nervous system relapse in DLBCL. These findings support integration of clinically informative AI-generated imaging metrics into clinical workflows to improve identification of high-risk DLBCL patients.

Trial Registration

Registered clinicaltrials.gov number: NCT01287741.

Graphical Abstract

Supplementary Information

The online version contains supplementary material available at 10.1186/s40644-022-00476-0.

Computer-aided detection and segmentation of malignant melanoma lesions on whole-body 18F-FDG PET/CT using an interpretable deep learning approach

BACKGROUND AND OBJECTIVE

In oncology, 18-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET) / computed tomography (CT) is widely used to identify and analyse metabolically-active tumours. The combination of the high sensitivity and specificity from 18F-FDG PET and the high resolution from CT makes accurate assessment of disease status and treatment response possible. Since cancer is a systemic disease, whole-body imaging is of high interest. Moreover, whole-body metabolic tumour burden is emerging as a promising new biomarker predicting outcome for innovative immunotherapy in different tumour types. However, this comes with certain challenges such as the large amount of data for manual reading, different appearance of lesions across the body and cumbersome reporting, hampering its use in clinical routine. Automation of the reading can facilitate the process, maximise the information retrieved from the images and support clinicians in making treatment decisions.

METHODS

This work proposes a fully automated system for lesion detection and segmentation on whole-body 18F-FDG PET/CT. The novelty of the method stems from the fact that the same two-step approach used when manually reading the images was adopted, consisting of an intensity-based thresholding on PET followed by a classification that specifies which regions represent normal physiological uptake and which are malignant tissue. The dataset contained 69 patients treated for malignant melanoma. Baseline and follow-up scans together offered 267 images for training and testing.

RESULTS

On an unseen dataset of 53 PET/CT images, a median F1-score of 0.7500 was achieved with, on average, 1.566 false positive lesions per scan. Metabolically-active tumours were segmented with a median dice score of 0.8493 and absolute volume difference of 0.2986 ml.

CONCLUSIONS

The proposed fully automated method for the segmentation and detection of metabolically-active lesions on whole-body 18F-FDG PET/CT achieved competitive results. Moreover, it was compared to a direct segmentation approach which it outperformed for all metrics.

Heart and bladder detection and segmentation on FDG PET/CT by deep learning

Purpose

Positron emission tomography (PET)/ computed tomography (CT) has been extensively used to quantify metabolically active tumors in various oncology indications. However, FDG-PET/CT often encounters false positives in tumor detection due to ¹⁸fluorodeoxyglucose (FDG) accumulation from the heart and bladder that often exhibit similar FDG uptake as tumors. Thus, it is necessary to eliminate this source of physiological noise. Major challenges for this task include: (1) large inter-patient variability in the appearance for the heart and bladder. (2) The size and shape of bladder or heart may appear different on PET and CT. (3) Tumors can be very close or connected to the heart or bladder.

Approach

A deep learning based approach is proposed to segment the heart and bladder on whole body PET/CT automatically. Two 3D U-Nets were developed separately to segment the heart and bladder, where each network receives the PET and CT as a multi-modal input. Data sets were obtained from retrospective clinical trials and include 575 PET/CT for heart segmentation and 538 for bladder segmentation.

Results

The models were evaluated on a test set from an independent trial and achieved a Dice Similarity Coefficient (DSC) of 0.96 for heart segmentation and 0.95 for bladder segmentation, Average Surface Distance (ASD) of 0.44 mm on heart and 0.90 mm on bladder.

Conclusions

This methodology could be a valuable component to the FDG-PET/CT data processing chain by removing FDG physiological noise associated with heart and/or bladder accumulation prior to image analysis by manual, semi- or automated tumor analysis methods.

Deep Learning Approach to Automatize TMTV Calculations Regardless of Segmentation Methodology for Major FDG-Avid Lymphomas

The total metabolic tumor volume (TMTV) is a new prognostic factor in lymphomas that could benefit from automation with deep learning convolutional neural networks (CNN). Manual TMTV segmentations of 1218 baseline 18FDG-PET/CT have been used for training. A 3D V-NET model has been trained to generate segmentations with soft dice loss. Ground truth segmentation has been generated using a combination of different thresholds (TMTVprob), applied to the manual region of interest (Otsu, relative 41% and SUV 2.5 and 4 cutoffs). In total, 407 and 405 PET/CT were used for test and validation datasets, respectively. The training was completed in 93 h. In comparison with the TMTVprob, mean dice reached 0.84 in the training set, 0.84 in the validation set and 0.76 in the test set. The median dice scores for each TMTV methodology were 0.77, 0.70 and 0.90 for 41%, 2.5 and 4 cutoff, respectively. Differences in the median TMTV between manual and predicted TMTV were 32, 147 and 5 mL. Spearman’s correlations between manual and predicted TMTV were 0.92, 0.95 and 0.98. This generic deep learning model to compute TMTV in lymphomas can drastically reduce computation time of TMTV.

Pre-treatment total metabolic tumour volumes in lymphoma: Does quantity matter?

Positron emission tomography/computed tomography (PET/CT) is used for the staging of lymphomas. Clinical information, such as Ann Arbor stage and number of involved sites, is derived from baseline staging and correlates with tumour volume. With modern imaging software, exact measures of total metabolic tumour volumes (tMTV) can be determined, in a semi- or fully-automated manner. Several technical factors, such as tumour segmentation and PET/CT technology influence tMTV and there is no consensus on a standardized uptake value (SUV) thresholding method, or how to include the volumes in the bone marrow and spleen. In diffuse large B-cell lymphoma, follicular lymphoma, peripheral T-cell lymphoma, and Hodgkin lymphoma, tMTV has been shown to predict progression-free survival and/or overall survival, after adjustments for clinical risk scores. However, most studies have used receiver operating curves to determine the optimal cut-off for tMTV and many studies did not include a training-validation approach, which led to the risk of overestimation of the independent prognostic value of tMTV. The identified cut-off values are heterogeneous, even when the same SUV thresholding method is used. Future studies should focus on testing tMTV in homogeneously-treated cohorts and seek to validate identified cut-off values externally so that a prognostic value can be documented, over and above currently used clinical surrogates for tumour volume.

Head and neck tumor segmentation in PET/CT: The HECKTOR challenge

This paper relates the post-analysis of the first edition of the HEad and neCK TumOR (HECKTOR) challenge. This challenge was held as a satellite event of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2020, and was the first of its kind focusing on lesion segmentation in combined FDG-PET and CT image modalities. The challenge's task is the automatic segmentation of the Gross Tumor Volume (GTV) of Head and Neck (H&N) oropharyngeal primary tumors in FDG-PET/CT images. To this end, the participants were given a training set of 201 cases from four different centers and their methods were tested on a held-out set of 53 cases from a fifth center. The methods were ranked according to the Dice Score Coefficient (DSC) averaged across all test cases. An additional inter-observer agreement study was organized to assess the difficulty of the task from a human perspective. 64 teams registered to the challenge, among which 10 provided a paper detailing their approach. The best method obtained an average DSC of 0.7591, showing a large improvement over our proposed baseline method and the inter-observer agreement, associated with DSCs of 0.6610 and 0.61, respectively. The automatic methods proved to successfully leverage the wealth of metabolic and structural properties of combined PET and CT modalities, significantly outperforming human inter-observer agreement level, semi-automatic thresholding based on PET images as well as other single modality-based methods. This promising performance is one step forward towards large-scale radiomics studies in H&N cancer, obviating the need for error-prone and time-consuming manual delineation of GTVs.

How molecular imaging will enable robotic precision surgery : The role of artificial intelligence, augmented reality, and navigation

Molecular imaging is one of the pillars of precision surgery. Its applications range from early diagnostics to therapy planning, execution, and the accurate assessment of outcomes. In particular, molecular imaging solutions are in high demand in minimally invasive surgical strategies, such as the substantially increasing field of robotic surgery. This review aims at connecting the molecular imaging and nuclear medicine community to the rapidly expanding armory of surgical medical devices. Such devices entail technologies ranging from artificial intelligence and computer-aided visualization technologies (software) to innovative molecular imaging modalities and surgical navigation (hardware). We discuss technologies based on their role at different steps of the surgical workflow, i.e., from surgical decision and planning, over to target localization and excision guidance, all the way to (back table) surgical verification. This provides a glimpse of how innovations from the technology fields can realize an exciting future for the molecular imaging and surgery communities.

Early molecular imaging response assessment based on determination of total viable tumor burden in [68Ga]Ga-PSMA-11 PET/CT independently predicts overall survival in [177Lu]Lu-PSMA-617 radioligand therapy

Purpose

In patients with metastatic castration-resistant prostate cancer (mCRPC) treated with prostate-specific membrane antigen-targeted radioligand therapy (PSMA-RLT), the predictive value of PSMA PET/CT-derived response is still under investigation. Early molecular imaging response based on total viable tumor burden and its association with overall survival (OS) was explored in this study.

Methods

Sixty-six mCRPC patients who received [¹⁷⁷Lu]Lu-PSMA-617 RLT within a prospective patient registry (REALITY Study, NCT04833517) were analyzed. Patients received a [⁶⁸Ga]Ga-PSMA-11 PET/CT scan before the first and after the second cycle of PSMA-RLT. Total lesion PSMA (TLP) was determined by semiautomatic whole-body tumor segmentation. Molecular imaging response was assessed by change in TLP and modified PERCIST criteria. Biochemical response was assessed using standard serum PSA and PCWG3 criteria. Both response assessment methods and additional baseline parameters were analyzed regarding their association with OS by univariate and multivariable analysis.

Results

By molecular imaging, 40/66 (60.6%) patients showed partial remission (PR), 19/66 (28.7%) stable disease (SD), and 7/66 (10.6%) progressive disease (PD). Biochemical response assessment revealed PR in 34/66 (51.5%) patients, SD in 20/66 (30.3%), and PD in 12/66 (18.2%). Response assessments were concordant in 49/66 (74.3%) cases. On univariate analysis, both molecular and biochemical response (p = 0.001 and 0.008, respectively) as well as two baseline characteristics (ALP and ECOG) were each significantly associated with OS. The median OS of patients showing molecular PR was 24.6 versus 10.7 months in the remaining patients (with SD or PD). On multivariable analysis molecular imaging response remained an independent predictor of OS (p = 0.002), eliminating biochemical response as insignificant (p = 0.515).

Conclusion

The new whole-body molecular imaging–derived biomarker, early change of total lesion PSMA (TLP), independently predicts overall survival in [¹⁷⁷Lu]Lu-PSMA-617 RLT in mCRPC, outperforming conventional PSA-based response assessment. TLP might be considered a more distinguished and advanced biomarker for monitoring PSMA-RLT over commonly used serum PSA.

Weakly supervised segmentation of tumor lesions in PET-CT hybrid imaging

Purpose: We introduce and evaluate deep learning methods for weakly supervised segmentation of tumor lesions in whole-body fluorodeoxyglucose-positron emission tomography (FDG-PET) based solely on binary global labels (“tumor” versus “no tumor”).

Approach: We propose a three-step approach based on (i) a deep learning framework for image classification, (ii) subsequent generation of class activation maps (CAMs) using different CAM methods (CAM, GradCAM, GradCAM++, ScoreCAM), and (iii) final tumor segmentation based on the aforementioned CAMs. A VGG-based classification neural network was trained to distinguish between PET image slices with and without FDG-avid tumor lesions. Subsequently, the CAMs of this network were used to identify the tumor regions within images. This proposed framework was applied to FDG-PET/CT data of 453 oncological patients with available manually generated ground-truth segmentations. Quantitative segmentation performance was assessed for the different CAM approaches and compared with the manual ground truth segmentation and with supervised segmentation methods. In addition, further biomarkers (MTV and TLG) were extracted from the segmentation masks.

Results: A weakly supervised segmentation of tumor lesions was feasible with satisfactory performance [best median Dice score 0.47, interquartile range (IQR) 0.35] compared with a fully supervised U-Net model (median Dice score 0.72, IQR 0.36) and a simple threshold based segmentation (Dice score 0.29, IQR 0.28). CAM, GradCAM++, and ScoreCAM yielded similar results. However, GradCAM led to inferior results (median Dice score: 0.12, IQR 0.21) and was likely to ignore multiple instances within a given slice. CAM, GradCAM++, and ScoreCAM yielded accurate estimates of metabolic tumor volume (MTV) and tumor lesion glycolysis. Again, worse results were observed for GradCAM.

Conclusions: This work demonstrated the feasibility of weakly supervised segmentation of tumor lesions and accurate estimation of derived metrics such as MTV and tumor lesion glycolysis.

FDG-PET/CT in Lymphoma: Where Do We Go Now?

18F-fluorodeoxyglucose positron emission tomography combined with computed tomography (FDG-PET/CT) is an essential part of the management of patients with lymphoma at staging and response evaluation. Efforts to standardize PET acquisition and reporting, including the 5-point Deauville scale, have enabled PET to become a surrogate for treatment success or failure in common lymphoma subtypes. This review summarizes the key clinical-trial evidence that supports PET-directed personalized approaches in lymphoma but also points out the potential place of innovative PET/CT metrics or new radiopharmaceuticals in the future.

Automated liver lesion detection in 68Ga DOTATATE PET/CT using a deep fully convolutional neural network

BACKGROUND

Gastroenteropancreatic neuroendocrine tumors most commonly metastasize to the liver; however, high normal background ⁶⁸Ga-DOTATATE activity and high image noise make metastatic lesions difficult to detect. The purpose of this study is to develop a rapid, automated and highly specific method to identify ⁶⁸Ga-DOTATATE PET/CT hepatic lesions using a 2D U-Net convolutional neural network.

METHODS

A retrospective study of ⁶⁸Ga-DOTATATE PET/CT patient studies (n = 125; 57 with ⁶⁸Ga-DOTATATE hepatic lesions and 68 without) was evaluated. The dataset was randomly divided into 75 studies for the training set (36 abnormal, 39 normal), 25 for the validation set (11 abnormal, 14 normal) and 25 for the testing set (11 abnormal, 14 normal). Hepatic lesions were physician annotated using a modified PERCIST threshold, and boundary definition by gradient edge detection. The 2D U-Net was trained independently five times for 100,000 iterations using a linear combination of binary cross-entropy and dice losses with a stochastic gradient descent algorithm. Performance metrics included: positive predictive value (PPV), sensitivity, F₁ score and area under the precision-recall curve (PR-AUC). Five different pixel area thresholds were used to filter noisy predictions.

RESULTS

A total of 233 lesions were annotated with each abnormal study containing a mean of 4 ± 2.75 lesions. A pixel filter of 20 produced the highest mean PPV 0.94 ± 0.01. A pixel filter of 5 produced the highest mean sensitivity 0.74 ± 0.02. The highest mean F₁ score 0.79 ± 0.01 was produced with a 20 pixel filter. The highest mean PR-AUC 0.73 ± 0.03 was produced with a 15 pixel filter.

CONCLUSION

Deep neural networks can automatically detect hepatic lesions in ⁶⁸Ga-DOTATATE PET. Ongoing improvements in data annotation methods, increasing sample sizes and training methods are anticipated to further improve detection performance.

Computer-aided detection of mantle cell lymphoma on 18F-FDG PET/CT using a deep learning convolutional neural network

¹⁸F-FDG PET/CT can provide quantitative characterization with prognostic value for mantle cell lymphoma (MCL). However, detection of MCL is performed manually, which is labor intensive and not a part of the routine clinical practice. This study investigates a deep learning convolutional neural network (DLCNN) for computer-aided detection of MCL on ¹⁸F-FDG PET/CT. We retrospectively analyzed 142 baseline ¹⁸F-FDG PET/CT scans of biopsy-confirmed MCL acquired between May 2007 and October 2018. Of the 142 scans, 110 were from our institution and 32 were from outside institutions. An Xception-based U-Net was constructed to classify each pixel of the PET/CT images as MCL or not. The network was first trained and tested on the within-institution scans by applying five-fold cross-validation. Sensitivity and false positives (FPs) per patient were calculated for network evaluation. The network was then tested on the outside-institution scans, which were excluded from network training. For the 110 within-institution patients (85 male; median age, 58 [range: 39-84] years), the network achieved an overall median sensitivity of 88% (interquartile range [IQR]: 25%) with 15 (IQR: 12) FPs/patient. Sensitivity was dependent on lesion size and SUV_max but not on lesion location. For the 32 outside-institution patients (24 male; median age, 59 [range: 40-67] years), the network achieved a median sensitivity of 84% (IQR: 24%) with 14 (IQR: 10) FPs/patient. No significant performance difference was found between the within and outside institution scans. Therefore, DLCNN can potentially help with MCL detection on ¹⁸F-FDG PET/CT with high sensitivity and limited FPs.

Deep learning convolutional neural network (DLCNN): unleashing the potential of 18F-FDG PET/CT in lymphoma

This perspective briefly reviewed the applications of 18F-FDG PET/CT in the clinical management of lymphoma and the need for lesion segmentation in those applications. It discussed the limitations of existing segmentation technologies and the great potential of using deep learning convolutional neural network (DLCNN) to accomplish automatic lymphoma segmentation and characterizations. Finally, the authors shared perspectives on the technical challenges that need to be addressed to fully unleash the potential of DLCNN and 18F-FDG PET/CT in the diagnosis, prognosis, and treatment of lymphoma.

New Approaches in Characterization of Lesions Dissemination in DLBCL Patients on Baseline PET/CT

Dissemination, expressed recently by the largest Euclidian distance between lymphoma sites (SDmax), appeared a promising risk factor in DLBCL patients. We investigated alternative distance metrics to characterize the robustness of the dissemination information. In 290 patients from the REMARC trial (NCT01122472), the Euclidean (Euc), Manhattan (Man), and Tchebychev (Tch) distances between the furthest lesions, firstly based on the centroid of each lesion and then directly from the two most distant tumor voxels and the Travelling Salesman Problem distance (TSP) were calculated. For PFS, the areas under the ROC curves were between 0.63 and 0.64, and between 0.62 and 0.65 for OS. Patients with high SDmax whatever the method of calculation or high SD_TSP had a significantly poorer outcome than patients with low SDmax or SD_TSP (p < 0.001 for both PFS and OS), with significance maintained in Ann Arbor advanced-stage patients. In multivariate analysis with total metabolic tumor volume and ECOG, each distance feature had an independent prognostic value for PFS. For OS, only SDmax_Tch, SDmax_Euc _Vox, and SDmax_Man _Vox reached significance. The spread of DLBCL lesions measured by the largest distance between lymphoma sites is a strong independent prognostic factor and could be measured directly from tumor voxels, allowing its development in the area of the deep learning segmentation methods.

Prediction of EGFR Mutation Status Based on 18F-FDG PET/CT Imaging Using Deep Learning-Based Model in Lung Adenocarcinoma

Objective

The purpose of this study was to develop a deep learning-based system to automatically predict epidermal growth factor receptor (EGFR) mutant lung adenocarcinoma in ¹⁸F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT).

Methods

Three hundred and one lung adenocarcinoma patients with EGFR mutation status were enrolled in this study. Two deep learning models (SE_CT and SE_PET) were developed with Squeeze-and-Excitation Residual Network (SE-ResNet) module for the prediction of EGFR mutation with CT and PET images, respectively. The deep learning models were trained with a training data set of 198 patients and tested with a testing data set of 103 patients. Stacked generalization was used to integrate the results of SE_CT and SE_PET.

Results

The AUCs of the SE_CT and SE_PET were 0.72 (95% CI, 0.62–0.80) and 0.74 (95% CI, 0.65–0.82) in the testing data set, respectively. After integrating SE_CT and SE_PET with stacked generalization, the AUC was further improved to 0.84 (95% CI, 0.75–0.90), significantly higher than SE_CT (p<0.05).

Conclusion

The stacking model based on ¹⁸F-FDG PET/CT images is capable to predict EGFR mutation status of patients with lung adenocarcinoma automatically and non-invasively. The proposed model in this study showed the potential to help clinicians identify suitable advanced patients with lung adenocarcinoma for EGFR‐targeted therapy.

Whole-body uptake classification and prostate cancer staging in 68Ga-PSMA-11 PET/CT using dual-tracer learning

Purpose

In PSMA-ligand PET/CT imaging, standardized evaluation frameworks and image-derived parameters are increasingly used to support prostate cancer staging. Clinical applicability remains challenging wherever manual measurements of numerous suspected lesions are required. Deep learning methods are promising for automated image analysis, typically requiring extensive expert-annotated image datasets to reach sufficient accuracy. We developed a deep learning method to support image-based staging, investigating the use of training information from two radiotracers.

Methods

In 173 subjects imaged with ⁶⁸Ga-PSMA-11 PET/CT, divided into development (121) and test (52) sets, we trained and evaluated a convolutional neural network to both classify sites of elevated tracer uptake as nonsuspicious or suspicious for cancer and assign them an anatomical location. We evaluated training strategies to leverage information from a larger dataset of ¹⁸F-FDG PET/CT images and expert annotations, including transfer learning and combined training encoding the tracer type as input to the network. We assessed the agreement between the N and M stage assigned based on the network annotations and expert annotations, according to the PROMISE miTNM framework.

Results

In the development set, including ¹⁸F-FDG training data improved classification performance in four-fold cross validation. In the test set, compared to expert assessment, training with ¹⁸F-FDG data and the development set yielded 80.4% average precision [confidence interval (CI): 71.1–87.8] for identification of suspicious uptake sites, 77% (CI: 70.0–83.4) accuracy for anatomical location classification of suspicious findings, 81% agreement for identification of regional lymph node involvement, and 77% agreement for identification of metastatic stage.

Conclusion

The evaluated algorithm showed good agreement with expert assessment for identification and anatomical location classification of suspicious uptake sites in whole-body ⁶⁸Ga-PSMA-11 PET/CT. With restricted PSMA-ligand data available, the use of training examples from a different radiotracer improved performance. The investigated methods are promising for enabling efficient assessment of cancer stage and tumor burden.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05473-2.

Tumor Segmentation in Patients with Head and Neck Cancers Using Deep Learning Based-on Multi-modality PET/CT Images

Segmentation of head and neck cancer (HNC) primary tumors onmedical images is an essential, yet labor-intensive, aspect of radiotherapy.PET/CT imaging offers a unique ability to capture metabolic and anatomicinformation, which is invaluable for tumor detection and border definition. Anautomatic segmentation tool that could leverage the dual streams of informationfrom PET and CT imaging simultaneously, could substantially propel HNCradiotherapy workflows forward. Herein, we leverage a multi-institutionalPET/CT dataset of 201 HNC patients, as part of the MICCAI segmentationchallenge, to develop novel deep learning architectures for primary tumor auto-segmentation for HNC patients. We preprocess PET/CT images by normalizingintensities and applying data augmentation to mitigate overfitting. Both 2D and3D convolutional neural networks based on the U-net architecture, which wereoptimized with a model loss function based on a combination of dice similaritycoefficient (DSC) and binary cross entropy, were implemented. The median andmean DSC values comparing the predicted tumor segmentation with the groundtruth achieved by the models through 5-fold cross validation are 0.79 and 0.69for the 3D model, respectively, and 0.79 and 0.67 for the 2D model, respec-tively. These promising results show potential to provide an automatic, accurate,and efficient approach for primary tumor auto-segmentation to improve theclinical practice of HNC treatment.

Artificial intelligence and hybrid imaging: the best match for personalized medicine in oncology

Artificial intelligence (AI) refers to a field of computer science aimed to perform tasks typically requiring human intelligence. Currently, AI is recognized in the broader technology radar within the five key technologies which emerge for their wide-ranging applications and impact in communities, companies, business, and value chain framework alike. However, AI in medical imaging is at an early phase of development, and there are still hurdles to take related to reliability, user confidence, and adoption. The present narrative review aimed to provide an overview on AI-based approaches (distributed learning, statistical learning, computer-aided diagnosis and detection systems, fully automated image analysis tool, natural language processing) in oncological hybrid medical imaging with respect to clinical tasks (detection, contouring and segmentation, prediction of histology and tumor stage, prediction of mutational status and molecular therapies targets, prediction of treatment response, and outcome). Particularly, AI-based approaches have been briefly described according to their purpose and, finally lung cancer-being one of the most extensively malignancy studied by hybrid medical imaging-has been used as illustrative scenario. Finally, we discussed clinical challenges and open issues including ethics, validation strategies, effective data-sharing methods, regulatory hurdles, educational resources, and strategy to facilitate the interaction among different stakeholders. Some of the major changes in medical imaging will come from the application of AI to workflow and protocols, eventually resulting in improved patient management and quality of life. Overall, several time-consuming tasks could be automatized. Machine learning algorithms and neural networks will permit sophisticated analysis resulting not only in major improvements in disease characterization through imaging, but also in the integration of multiple-omics data (i.e., derived from pathology, genomic, proteomics, and demographics) for multi-dimensional disease featuring. Nevertheless, to accelerate the transition of the theory to practice a sustainable development plan considering the multi-dimensional interactions between professionals, technology, industry, markets, policy, culture, and civil society directed by a mindset which will allow talents to thrive is necessary.