Comparison of 11 automated PET segmentation methods in lymphoma

TMTV-Net: fully automated total metabolic tumor volume segmentation in lymphoma PET/CT images - a multi-center generalizability analysis

PURPOSE

Total metabolic tumor volume (TMTV) segmentation has significant value enabling quantitative imaging biomarkers for lymphoma management. In this work, we tackle the challenging task of automated tumor delineation in lymphoma from PET/CT scans using a cascaded approach.

METHODS

Our study included 1418 2-[18F]FDG PET/CT scans from four different centers. The dataset was divided into 900 scans for development/validation/testing phases and 518 for multi-center external testing. The former consisted of 450 lymphoma, lung cancer, and melanoma scans, along with 450 negative scans, while the latter consisted of lymphoma patients from different centers with diffuse large B cell, primary mediastinal large B cell, and classic Hodgkin lymphoma cases. Our approach involves resampling PET/CT images into different voxel sizes in the first step, followed by training multi-resolution 3D U-Nets on each resampled dataset using a fivefold cross-validation scheme. The models trained on different data splits were ensemble. After applying soft voting to the predicted masks, in the second step, we input the probability-averaged predictions, along with the input imaging data, into another 3D U-Net. Models were trained with semi-supervised loss. We additionally considered the effectiveness of using test time augmentation (TTA) to improve the segmentation performance after training. In addition to quantitative analysis including Dice score (DSC) and TMTV comparisons, the qualitative evaluation was also conducted by nuclear medicine physicians.

RESULTS

Our cascaded soft-voting guided approach resulted in performance with an average DSC of 0.68 ± 0.12 for the internal test data from developmental dataset, and an average DSC of 0.66 ± 0.18 on the multi-site external data (n = 518), significantly outperforming (p < 0.001) state-of-the-art (SOTA) approaches including nnU-Net and SWIN UNETR. While TTA yielded enhanced performance gains for some of the comparator methods, its impact on our cascaded approach was found to be negligible (DSC: 0.66 ± 0.16). Our approach reliably quantified TMTV, with a correlation of 0.89 with the ground truth (p < 0.001). Furthermore, in terms of visual assessment, concordance between quantitative evaluations and clinician feedback was observed in the majority of cases. The average relative error (ARE) and the absolute error (AE) in TMTV prediction on external multi-centric dataset were ARE = 0.43 ± 0.54 and AE = 157.32 ± 378.12 (mL) for all the external test data (n = 518), and ARE = 0.30 ± 0.22 and AE = 82.05 ± 99.78 (mL) when the 10% outliers (n = 53) were excluded.

CONCLUSION

TMTV-Net demonstrates strong performance and generalizability in TMTV segmentation across multi-site external datasets, encompassing various lymphoma subtypes. A negligible reduction of 2% in overall performance during testing on external data highlights robust model generalizability across different centers and cancer types, likely attributable to its training with resampled inputs. Our model is publicly available, allowing easy multi-site evaluation and generalizability analysis on datasets from different institutions.

Metabolic tumour volume in Hodgkin lymphoma-A comparison between manual and AI-based analysis

AIM

To compare total metabolic tumour volume (tMTV), calculated using two artificial intelligence (AI)-based tools, with manual segmentation by specialists as the reference.

METHODS

Forty-eight consecutive Hodgkin lymphoma (HL) patients staged with [18F] fluorodeoxyglucose positron emission tomography/computed tomography were included. The median age was 35 years (range: 7-75), 46% female. The tMTV was automatically measured using the AI-based tools positron emission tomography assisted reporting system (PARS) (from Siemens) and RECOMIA (recomia.org) without any manual adjustments. A group of eight nuclear medicine specialists manually segmented lesions for tMTV calculations; each patient was independently segmented by two specialists.

RESULTS

The median of the manual tMTV was 146 cm3 (interquartile range [IQR]: 79-568 cm3) and the median difference between two tMTV values segmented by different specialists for the same patient was 26 cm3 (IQR: 10-86 cm3). In 22 of the 48 patients, the manual tMTV value was closer to the RECOMIA tMTV value than to the manual tMTV value segmented by the second specialist. In 11 of the remaining 26 patients, the difference between the RECOMIA tMTV and the manual tMTV was small (<26 cm3, which was the median difference between two manual tMTV values from the same patient). The corresponding numbers for PARS were 18 and 10 patients, respectively.

CONCLUSION

The results of this study indicate that RECOMIA and Siemens PARS AI tools could be used without any major manual adjustments in 69% (33/48) and 58% (28/48) of HL patients, respectively. This demonstrates the feasibility of using AI tools to support physicians measuring tMTV for assessment of prognosis in clinical practice.

Quantitative PET-based biomarkers in lymphoma: getting ready for primetime

The use of functional quantitative biomarkers extracted from routine PET-CT scans to characterize clinical responses in patients with lymphoma is gaining increased attention, and these biomarkers can outperform established clinical risk factors. Total metabolic tumour volume enables individualized estimation of survival outcomes in patients with lymphoma and has shown the potential to predict response to therapy suitable for  risk-adapted treatment approaches in clinical trials. The deployment of machine learning tools in molecular imaging research can assist in recognizing complex patterns and, with image classification, in tumour identification and segmentation of data from PET-CT scans. Initial studies using fully automated approaches to calculate metabolic tumour volume and other PET-based biomarkers have demonstrated appropriate correlation with calculations from experts, warranting further testing in large-scale studies. The extraction of computer-based quantitative tumour characterization through radiomics can provide a comprehensive view of phenotypic heterogeneity that better captures the molecular and functional features of the disease. Additionally, radiomics can be integrated with genomic data to provide more accurate prognostic information. Further improvements in PET-based biomarkers are imminent, although their incorporation into clinical decision-making currently has methodological shortcomings that need to be addressed with confirmatory prospective validation in selected patient populations. In this Review, we discuss the current knowledge, challenges and opportunities in the integration of quantitative PET-based biomarkers in clinical trials and the routine management of patients with lymphoma.

Evaluation of Semiautomatic and Deep Learning-Based Fully Automatic Segmentation Methods on [18F]FDG PET/CT Images from Patients with Lymphoma: Influence on Tumor Characterization

The objective is to assess the performance of seven semiautomatic and two fully automatic segmentation methods on [18F]FDG PET/CT lymphoma images and evaluate their influence on tumor quantification. All lymphoma lesions identified in 65 whole-body [18F]FDG PET/CT staging images were segmented by two experienced observers using manual and semiautomatic methods. Semiautomatic segmentation using absolute and relative thresholds, k-means and Bayesian clustering, and a self-adaptive configuration (SAC) of k-means and Bayesian was applied. Three state-of-the-art deep learning-based segmentations methods using a 3D U-Net architecture were also applied. One was semiautomatic and two were fully automatic, of which one is publicly available. Dice coefficient (DC) measured segmentation overlap, considering manual segmentation the ground truth. Lymphoma lesions were characterized by 31 features. Intraclass correlation coefficient (ICC) assessed features agreement between different segmentation methods. Nine hundred twenty [18F]FDG-avid lesions were identified. The SAC Bayesian method achieved the highest median intra-observer DC (0.87). Inter-observers' DC was higher for SAC Bayesian than manual segmentation (0.94 vs 0.84, p < 0.001). Semiautomatic deep learning-based median DC was promising (0.83 (Obs1), 0.79 (Obs2)). Threshold-based methods and publicly available 3D U-Net gave poorer results (0.56 ≤ DC ≤ 0.68). Maximum, mean, and peak standardized uptake values, metabolic tumor volume, and total lesion glycolysis showed excellent agreement (ICC ≥ 0.92) between manual and SAC Bayesian segmentation methods. The SAC Bayesian classifier is more reproducible and produces similar lesion features compared to manual segmentation, giving the best concordant results of all other methods. Deep learning-based segmentation can achieve overall good segmentation results but failed in few patients impacting patients' clinical evaluation.

Semi-automated 18F-FDG PET segmentation methods for tumor volume determination in Non-Hodgkin lymphoma patients: a literature review, implementation and multi-threshold evaluation

In the treatment of Non-Hodgkin lymphoma (NHL), multiple therapeutic options are available. Improving outcome predictions are essential to optimize treatment. The metabolic active tumor volume (MATV) has shown to be a prognostic factor in NHL. It is usually retrieved using semi-automated thresholding methods based on standardized uptake values (SUV), calculated from ¹⁸F-Fluorodeoxyglucose Positron Emission Tomography (¹⁸F-FDG PET) images. However, there is currently no consensus method for NHL. The aim of this study was to review literature on different segmentation methods used, and to evaluate selected methods by using an in house created software tool. A software tool, MUltiple SUV Threshold (MUST)-segmenter was developed where tumor locations are identified by placing seed-points on the PET images, followed by subsequent region growing. Based on a literature review, 9 SUV thresholding methods were selected and MATVs were extracted. The MUST-segmenter was utilized in a cohort of 68 patients with NHL. Differences in MATVs were assessed with paired t-tests, and correlations and distributions figures. High variability and significant differences between the MATVs based on different segmentation methods (p < 0.05) were observed in the NHL patients. Median MATVs ranged from 35 to 211 cc. No consensus for determining MATV is available based on the literature. Using the MUST-segmenter with 9 selected SUV thresholding methods, we demonstrated a large and significant variation in MATVs. Identifying the most optimal segmentation method for patients with NHL is essential to further improve predictions of toxicity, response, and treatment outcomes, which can be facilitated by the MUST-segmenter.

The application of machine learning to imaging in hematological oncology: A scoping review

Background

Here, we conducted a scoping review to (i) establish which machine learning (ML) methods have been applied to hematological malignancy imaging; (ii) establish how ML is being applied to hematological cancer radiology; and (iii) identify addressable research gaps.

Methods

The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews guidelines. The inclusion criteria were (i) pediatric and adult patients with suspected or confirmed hematological malignancy undergoing imaging (population); (ii) any study using ML techniques to derive models using radiological images to apply to the clinical management of these patients (concept); and (iii) original research articles conducted in any setting globally (context). Quality Assessment of Diagnostic Accuracy Studies 2 criteria were used to assess diagnostic and segmentation studies, while the Newcastle-Ottawa scale was used to assess the quality of observational studies.

Results

Of 53 eligible studies, 33 applied diverse ML techniques to diagnose hematological malignancies or to differentiate them from other diseases, especially discriminating gliomas from primary central nervous system lymphomas (n=18); 11 applied ML to segmentation tasks, while 9 applied ML to prognostication or predicting therapeutic responses, especially for diffuse large B-cell lymphoma. All studies reported discrimination statistics, but no study calculated calibration statistics. Every diagnostic/segmentation study had a high risk of bias due to their case-control design; many studies failed to provide adequate details of the reference standard; and only a few studies used independent validation.

Conclusion

To deliver validated ML-based models to radiologists managing hematological malignancies, future studies should (i) adhere to standardized, high-quality reporting guidelines such as the Checklist for Artificial Intelligence in Medical Imaging; (ii) validate models in independent cohorts; (ii) standardize volume segmentation methods for segmentation tasks; (iv) establish comprehensive prospective studies that include different tumor grades, comparisons with radiologists, optimal imaging modalities, sequences, and planes; (v) include side-by-side comparisons of different methods; and (vi) include low- and middle-income countries in multicentric studies to enhance generalizability and reduce inequity.

Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development

The nuclear medicine field has seen a rapid expansion of academic and commercial interest in developing artificial intelligence (AI) algorithms. Users and developers can avoid some of the pitfalls of AI by recognizing and following best practices in AI algorithm development. In this article, recommendations on technical best practices for developing AI algorithms in nuclear medicine are provided, beginning with general recommendations and then continuing with descriptions of how one might practice these principles for specific topics within nuclear medicine. This report was produced by the AI Task Force of the Society of Nuclear Medicine and Molecular Imaging.

Artificial Intelligence in Lymphoma PET Imaging:: A Scoping Review (Current Trends and Future Directions)

Malignant lymphomas are a family of heterogenous disorders caused by clonal proliferation of lymphocytes. 18F-FDG-PET has proven to provide essential information for accurate quantification of disease burden, treatment response evaluation, and prognostication. However, manual delineation of hypermetabolic lesions is often a time-consuming and impractical task. Applications of artificial intelligence (AI) may provide solutions to overcome this challenge. Beyond segmentation and detection of lesions, AI could enhance tumor characterization and heterogeneity quantification, as well as treatment response prediction and recurrence risk stratification. In this scoping review, we have systematically mapped and discussed the current applications of AI (such as detection, classification, segmentation as well as the prediction and prognostication) in lymphoma PET.

Trustworthy Artificial Intelligence in Medical Imaging

Trust in artificial intelligence (AI) by society and the development of trustworthy AI systems and ecosystems are critical for the progress and implementation of AI technology in medicine. With the growing use of AI in a variety of medical and imaging applications, it is more vital than ever to make these systems dependable and trustworthy. Fourteen core principles are considered in this article aiming to move the needle more closely to systems that are accurate, resilient, fair, explainable, safe, and transparent: toward trustworthy AI.

Outcome-based multiobjective optimization of lymphoma radiation therapy plans

At its core, radiation therapy (RT) requires balancing therapeutic effects against risk of adverse events in cancer survivors. The radiation oncologist weighs numerous disease and patient-level factors when considering the expected risk-benefit ratio of combined treatment modalities. As part of this, RT plan optimization software is used to find a clinically acceptable RT plan delivering a prescribed dose to the target volume while respecting pre-defined radiation dose-volume constraints for selected organs at risk. The obvious limitation to the current approach is that it is virtually impossible to ensure the selected treatment plan could not be bettered by an alternative plan providing improved disease control and/or reduced risk of adverse events in this individual. Outcome-based optimization refers to a strategy where all planning objectives are defined by modeled estimates of a specific outcome's probability. Noting that various adverse events and disease control are generally incommensurable, leads to the concept of a Pareto-optimal plan: a plan where no single objective can be improved without degrading one or more of the remaining objectives. Further benefits of outcome-based multiobjective optimization are that quantitative estimates of risks and benefit are obtained as are the effects of choosing a different trade-off between competing objectives. Furthermore, patient-level risk factors and combined treatment modalities may be integrated directly into plan optimization. Here, we present this approach in the clinical setting of multimodality therapy for malignant lymphoma, a malignancy with marked heterogeneity in biology, target localization, and patient characteristics. We discuss future research priorities including the potential of artificial intelligence.

Toward High-Throughput Artificial Intelligence-Based Segmentation in Oncological PET Imaging

Artificial intelligence (AI) techniques for image-based segmentation have garnered much attention in recent years. Convolutional neural networks have shown impressive results and potential toward fully automated segmentation in medical imaging, and particularly PET imaging. To cope with the limited access to annotated data needed in supervised AI methods, given tedious and prone-to-error manual delineations, semi-supervised and unsupervised AI techniques have also been explored for segmentation of tumors or normal organs in single- and bimodality scans. This work reviews existing AI techniques for segmentation tasks and the evaluation criteria for translational AI-based segmentation efforts toward routine adoption in clinical workflows.

Whole-body uptake classification and prostate cancer staging in 68Ga-PSMA-11 PET/CT using dual-tracer learning

Purpose

In PSMA-ligand PET/CT imaging, standardized evaluation frameworks and image-derived parameters are increasingly used to support prostate cancer staging. Clinical applicability remains challenging wherever manual measurements of numerous suspected lesions are required. Deep learning methods are promising for automated image analysis, typically requiring extensive expert-annotated image datasets to reach sufficient accuracy. We developed a deep learning method to support image-based staging, investigating the use of training information from two radiotracers.

Methods

In 173 subjects imaged with ⁶⁸Ga-PSMA-11 PET/CT, divided into development (121) and test (52) sets, we trained and evaluated a convolutional neural network to both classify sites of elevated tracer uptake as nonsuspicious or suspicious for cancer and assign them an anatomical location. We evaluated training strategies to leverage information from a larger dataset of ¹⁸F-FDG PET/CT images and expert annotations, including transfer learning and combined training encoding the tracer type as input to the network. We assessed the agreement between the N and M stage assigned based on the network annotations and expert annotations, according to the PROMISE miTNM framework.

Results

In the development set, including ¹⁸F-FDG training data improved classification performance in four-fold cross validation. In the test set, compared to expert assessment, training with ¹⁸F-FDG data and the development set yielded 80.4% average precision [confidence interval (CI): 71.1–87.8] for identification of suspicious uptake sites, 77% (CI: 70.0–83.4) accuracy for anatomical location classification of suspicious findings, 81% agreement for identification of regional lymph node involvement, and 77% agreement for identification of metastatic stage.

Conclusion

The evaluated algorithm showed good agreement with expert assessment for identification and anatomical location classification of suspicious uptake sites in whole-body ⁶⁸Ga-PSMA-11 PET/CT. With restricted PSMA-ligand data available, the use of training examples from a different radiotracer improved performance. The investigated methods are promising for enabling efficient assessment of cancer stage and tumor burden.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05473-2.

Interobserver agreement in automated metabolic tumor volume measurements of Deauville score 4 and 5 lesions at interim 18F-FDG PET in DLBCL

Introduction: Metabolic tumor volume (MTV) on interim-PET (I-PET) is a potential prognostic biomarker for diffuse large B-cell lymphoma (DLBCL). Implementation of MTV on I-PET requires consensus which semi-automated segmentation method delineates lesions most successfully with least user interaction. Methods used for baseline PET are not necessarily optimal for I-PET due to lower lesional standardized uptake values (SUV) at I-PET. Therefore, we aimed to evaluate which method provides the best delineation quality of Deauville-score (DS) 4-5 DLBCL lesions on I-PET at best interobserver agreement on delineation quality and, secondly, to assess the effect of lesional SUV_max on delineation quality and performance agreements. Methods: DS4-5 lesions from 45 I-PET scans were delineated using six semi-automated methods i) SUV 2.5, ii) SUV 4.0, iii) adaptive threshold [A50%peak], iv) 41% of maximum SUV [41%max], v) majority vote including voxels detected by ≥2 methods [MV2] and vi) detected by ≥3 methods [MV3]. Delineation quality per MTV was rated by three independent observers as acceptable or non-acceptable. For each method, observer scores on delineation quality, specific agreements and MTV were assessed for all lesions, and per category of lesional SUV_max (<5, 5-10, >10). Results: In 60 DS4-5 lesions on I-PET, MV3 performed best, with acceptable delineation in 90% of lesions, with a positive agreement (PA) of 93%. Delineation quality scores and agreements per method strongly depended on lesional SUV: the best delineation quality scores were obtained using MV3 in lesions with SUV_max<10 and SUV4.0 in more FDG-avid lesions. Consequently, overall delineation quality and PA improved by applying the most preferred method per SUV category instead of using MV3 as single best method. MV3- and SUV4.0-derived MTVs of lesions with SUV_max>10, were comparable after excluding visually failed MV3 contouring. For lesions with SUV_max<10, MTVs using different methods correlated poorly. Conclusion: On I-PET, MV3 performed best and provided the highest interobserver agreement regarding acceptable delineations of DS4-5 DLBCL lesions. However, delineation method preference strongly depended on lesional SUV. Therefore, we suggest to explore an approach that identifies the optimal delineation method per lesion as function of tumor FDG uptake characteristics, i.e. SUV_max.