Quality assurance for automatically generated contours with additional deep learning

Artificial intelligence uncertainty quantification in radiotherapy applications - A scoping review

BACKGROUND/PURPOSE

The use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions.

METHODS

We followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics.

RESULTS

We identified 56 articles published from 2015 to 2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50 %), followed by image-synthesis (13 %), and multiple applications simultaneously (11 %). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32 %). Imaging data was used in 91 % of studies, while only 13 % incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60 %), with Monte Carlo dropout being the most commonly implemented UQ method (32 %) followed by ensembling (16 %). 55 % of studies did not share code or datasets.

CONCLUSION

Our review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, we identified a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.

Deep learning-based automatic contour quality assurance for auto-segmented abdominal MR-Linac contours

Objective.Deep-learning auto-segmentation (DLAS) aims to streamline contouring in clinical settings. Nevertheless, achieving clinical acceptance of DLAS remains a hurdle in abdominal MRI, hindering the implementation of efficient clinical workflows for MR-guided online adaptive radiotherapy (MRgOART). Integrating automated contour quality assurance (ACQA) with automatic contour correction (ACC) techniques could optimize the performance of ACC by concentrating on inaccurate contours. Furthermore, ACQA can facilitate the contour selection process from various DLAS tools and/or deformable contour propagation from a prior treatment session. Here, we present the performance of novel DL-based 3D ACQA models for evaluating DLAS contours acquired during MRgOART.Approach.The ACQA model, based on a 3D convolutional neural network (CNN), was trained using pancreas and duodenum contours obtained from a research DLAS tool on abdominal MRIs acquired from a 1.5 T MR-Linac. The training dataset contained abdominal MR images, DL contours, and their corresponding quality ratings, from 103 datasets. The quality of DLAS contours was determined using an in-house contour classification tool, which categorizes contours as acceptable or edit-required based on the expected editing effort. The performance of the 3D ACQA model was evaluated using an independent dataset of 34 abdominal MRIs, utilizing confusion matrices for true and predicted classes.Main results.The ACQA predicted 'acceptable' and 'edit-required' contours at 72.2% (91/126) and 83.6% (726/868) accuracy for pancreas, and at 71.2% (79/111) and 89.6% (772/862) for duodenum contours, respectively. The model successfully identified false positive (extra) and false negative (missing) DLAS contours at 93.75% (15/16) and %99.7 (438/439) accuracy for pancreas, and at 95% (57/60) and 98.9% (91/99) for duodenum, respectively.Significance.We developed 3D-ACQA models capable of quickly evaluating the quality of DLAS pancreas and duodenum contours on abdominal MRI. These models can be integrated into clinical workflow, facilitating efficient and consistent contour evaluation process in MRgOART for abdominal malignancies.

Can we predict pathology without surgery? Weighing the added value of multiparametric MRI and whole prostate radiomics in integrative machine learning models

OBJECTIVE

To test the ability of high-performance machine learning (ML) models employing clinical, radiological, and radiomic variables to improve non-invasive prediction of the pathological status of prostate cancer (PCa) in a large, single-institution cohort.

METHODS

Patients who underwent multiparametric MRI and prostatectomy in our institution in 2015-2018 were considered; a total of 949 patients were included. Gradient-boosted decision tree models were separately trained using clinical features alone and in combination with radiological reporting and/or prostate radiomic features to predict pathological T, pathological N, ISUP score, and their change from preclinical assessment. Model behavior was analyzed in terms of performance, feature importance, Shapley additive explanation (SHAP) values, and mean absolute error (MAE). The best model was compared against a naïve model mimicking clinical workflow.

RESULTS

The model including all variables was the best performing (AUC values ranging from 0.73 to 0.96 for the six endpoints). Radiomic features brought a small yet measurable boost in performance, with the SHAP values indicating that their contribution can be critical to successful prediction of endpoints for individual patients. MAEs were lower for low-risk patients, suggesting that the models find them easier to classify. The best model outperformed (p ≤ 0.0001) clinical baseline, resulting in significantly fewer false negative predictions and overall was less prone to under-staging.

CONCLUSIONS

Our results highlight the potential benefit of integrative ML models for pathological status prediction in PCa. Additional studies regarding clinical integration of such models can provide valuable information for personalizing therapy offering a tool to improve non-invasive prediction of pathological status.

CLINICAL RELEVANCE STATEMENT

The best machine learning model was less prone to under-staging of the disease. The improved accuracy of our pathological prediction models could constitute an asset to the clinical workflow by providing clinicians with accurate pathological predictions prior to treatment.

KEY POINTS

• Currently, the most common strategies for pre-surgical stratification of prostate cancer (PCa) patients have shown to have suboptimal performances. • The addition of radiological features to the clinical features gave a considerable boost in model performance. Our best model outperforms the naïve model, avoiding under-staging and resulting in a critical advantage in the clinic. •Machine learning models incorporating clinical, radiological, and radiomics features significantly improved accuracy of pathological prediction in prostate cancer, possibly constituting an asset to the clinical workflow.

Automatic gross tumor volume segmentation with failure detection for safe implementation in locally advanced cervical cancer

Background and Purpose

Automatic segmentation methods have greatly changed the RadioTherapy (RT) workflow, but still need to be extended to target volumes. In this paper, Deep Learning (DL) models were compared for Gross Tumor Volume (GTV) segmentation in locally advanced cervical cancer, and a novel investigation into failure detection was introduced by utilizing radiomic features.

Methods and materials

We trained eight DL models (UNet, VNet, SegResNet, SegResNetVAE) for 2D and 3D segmentation. Ensembling individually trained models during cross-validation generated the final segmentation. To detect failures, binary classifiers were trained using radiomic features extracted from segmented GTVs as inputs, aiming to classify contours based on whether their Dice Similarity Coefficient ( DSC ) < T and DSC ⩾ T . Two distinct cohorts of T2-Weighted (T2W) pre-RT MR images captured in 2D sequences were used: one retrospective cohort consisting of 115 LACC patients from 30 scanners, and the other prospective cohort, comprising 51 patients from 7 scanners, used for testing.

Results

Segmentation by 2D-SegResNet achieved the best DSC, Surface DSC (SDSC 3 mm ), and 95th Hausdorff Distance (95HD): DSC = 0.72 ± 0.16,SDSC 3 mm =0.66 ± 0.17, and 95HD = 14.6 ± 9.0 mm without missing segmentation ( M =0) on the test cohort. Failure detection could generate precision ( P = 0.88 ), recall ( R = 0.75 ), F1-score ( F = 0.81 ), and accuracy ( A = 0.86 ) using Logistic Regression (LR) classifier on the test cohort with a threshold T = 0.67 on DSC values.

Conclusions

Our study revealed that segmentation accuracy varies slightly among different DL methods, with 2D networks outperforming 3D networks in 2D MRI sequences. Doctors found the time-saving aspect advantageous. The proposed failure detection could guide doctors in sensitive cases.

A network score-based metric to optimize the quality assurance of automatic radiotherapy target segmentations

Background and purpose

Existing methods for quality assurance of the radiotherapy auto-segmentations focus on the correlation between the average model entropy and the Dice Similarity Coefficient (DSC) only. We identified a metric directly derived from the output of the network and correlated it with clinically relevant metrics for contour accuracy.

Materials and Methods

Magnetic Resonance Imaging auto-segmentations were available for the gross tumor volume for cervical cancer brachytherapy (106 segmentations) and for the clinical target volume for rectal cancer external-beam radiotherapy (77 segmentations). The nnU-Net's output before binarization was taken as a score map. We defined a metric as the mean of the voxels in the score map above a threshold (λ). Comparisons were made with the mean and standard deviation over the score map and with the mean over the entropy map. The DSC, the 95th Hausdorff distance, the mean surface distance (MSD) and the surface DSC were computed for segmentation quality. Correlations between the studied metrics and model quality were assessed with the Pearson correlation coefficient (r). The area under the curve (AUC) was determined for detecting segmentations that require reviewing.

Results

For both tasks, our metric (λ = 0.30) correlated more strongly with the segmentation quality than the mean over the entropy map (for surface DSC, r > 0.65 vs. r < 0.60). The AUC was above 0.84 for detecting MSD values above 2 mm.

Conclusions

Our metric correlated strongly with clinically relevant segmentation metrics and detected segmentations that required reviewing, indicating its potential for automatic quality assurance of radiotherapy target auto-segmentations.

Deep-Learning-Based Dose Predictor for Glioblastoma-Assessing the Sensitivity and Robustness for Dose Awareness in Contouring

External beam radiation therapy requires a sophisticated and laborious planning procedure. To improve the efficiency and quality of this procedure, machine-learning models that predict these dose distributions were introduced. The most recent dose prediction models are based on deep-learning architectures called 3D U-Nets that give good approximations of the dose in 3D almost instantly. Our purpose was to train such a 3D dose prediction model for glioblastoma VMAT treatment and test its robustness and sensitivity for the purpose of quality assurance of automatic contouring. From a cohort of 125 glioblastoma (GBM) patients, VMAT plans were created according to a clinical protocol. The initial model was trained on a cascaded 3D U-Net. A total of 60 cases were used for training, 15 for validation and 20 for testing. The prediction model was tested for sensitivity to dose changes when subject to realistic contour variations. Additionally, the model was tested for robustness by exposing it to a worst-case test set containing out-of-distribution cases. The initially trained prediction model had a dose score of 0.94 Gy and a mean DVH (dose volume histograms) score for all structures of 1.95 Gy. In terms of sensitivity, the model was able to predict the dose changes that occurred due to the contour variations with a mean error of 1.38 Gy. We obtained a 3D VMAT dose prediction model for GBM with limited data, providing good sensitivity to realistic contour variations. We tested and improved the model's robustness by targeted updates to the training set, making it a useful technique for introducing dose awareness in the contouring evaluation and quality assurance process.

Comparison of automated segmentation techniques for magnetic resonance images of the prostate

BACKGROUND

Contouring of anatomical regions is a crucial step in the medical workflow and is both time-consuming and prone to intra- and inter-observer variability. This study compares different strategies for automatic segmentation of the prostate in T2-weighted MRIs.

METHODS

This study included 100 patients diagnosed with prostate adenocarcinoma who had undergone multi-parametric MRI and prostatectomy. From the T2-weighted MR images, ground truth segmentation masks were established by consensus from two expert radiologists. The prostate was then automatically contoured with six different methods: (1) a multi-atlas algorithm, (2) a proprietary algorithm in the Syngo.Via medical imaging software, and four deep learning models: (3) a V-net trained from scratch, (4) a pre-trained 2D U-net, (5) a GAN extension of the 2D U-net, and (6) a segmentation-adapted EfficientDet architecture. The resulting segmentations were compared and scored against the ground truth masks with one 70/30 and one 50/50 train/test data split. We also analyzed the association between segmentation performance and clinical variables.

RESULTS

The best performing method was the adapted EfficientDet (model 6), achieving a mean Dice coefficient of 0.914, a mean absolute volume difference of 5.9%, a mean surface distance (MSD) of 1.93 pixels, and a mean 95th percentile Hausdorff distance of 3.77 pixels. The deep learning models were less prone to serious errors (0.854 minimum Dice and 4.02 maximum MSD), and no significant relationship was found between segmentation performance and clinical variables.

CONCLUSIONS

Deep learning-based segmentation techniques can consistently achieve Dice coefficients of 0.9 or above with as few as 50 training patients, regardless of architectural archetype. The atlas-based and Syngo.via methods found in commercial clinical software performed significantly worse (0.855[Formula: see text]0.887 Dice).