Machine learning-based detection of aberrant deep learning segmentations of target and organs at risk for prostate radiotherapy using a secondary segmentation algorithm

NRG Oncology Assessment of Artificial Intelligence Deep Learning-Based Auto-segmentation for Radiation Therapy: Current Developments, Clinical Considerations, and Future Directions

Deep learning neural networks (DLNN) in Artificial intelligence (AI) have been extensively explored for automatic segmentation in radiotherapy (RT). In contrast to traditional model-based methods, data-driven AI-based models for auto-segmentation have shown high accuracy in early studies in research settings and controlled environment (single institution). Vendor-provided commercial AI models are made available as part of the integrated treatment planning system (TPS) or as a stand-alone tool that provides streamlined workflow interacting with the main TPS. These commercial tools have drawn clinics' attention thanks to their significant benefit in reducing the workload from manual contouring and shortening the duration of treatment planning. However, challenges occur when applying these commercial AI-based segmentation models to diverse clinical scenarios, particularly in uncontrolled environments. Contouring nomenclature and guideline standardization has been the main task undertaken by the NRG Oncology. AI auto-segmentation holds the potential clinical trial participants to reduce interobserver variations, nomenclature non-compliance, and contouring guideline deviations. Meanwhile, trial reviewers could use AI tools to verify contour accuracy and compliance of those submitted datasets. In recognizing the growing clinical utilization and potential of these commercial AI auto-segmentation tools, NRG Oncology has formed a working group to evaluate the clinical utilization and potential of commercial AI auto-segmentation tools. The group will assess in-house and commercially available AI models, evaluation metrics, clinical challenges, and limitations, as well as future developments in addressing these challenges. General recommendations are made in terms of the implementation of these commercial AI models, as well as precautions in recognizing the challenges and limitations.

A multi-modal vision-language pipeline strategy for contour quality assurance and adaptive optimization

Objective.Accurate delineation of organs-at-risk (OARs) is a critical step in radiotherapy. The deep learning generated segmentations usually need to be reviewed and corrected by oncologists manually, which is time-consuming and operator-dependent. Therefore, an automated quality assurance (QA) and adaptive optimization correction strategy was proposed to identify and optimize 'incorrect' auto-segmentations.Approach.A total of 586 CT images and labels from nine institutions were used. The OARs included the brainstem, parotid, and mandible. The deep learning generated contours were compared with the manual ground truth delineations. In this study, we proposed a novel contour quality assurance and adaptive optimization (CQA-AO) strategy, which consists of the following three main components: (1) the contour QA module classified the deep learning generated contours as either accepted or unaccepted; (2) the unacceptable contour categories analysis module provided the potential error reasons (five unacceptable category) and locations (attention heatmaps); (3) the adaptive correction of unacceptable contours module integrate vision-language representations and utilize convex optimization algorithms to achieve adaptive correction of 'incorrect' contours.Main results. In the contour QA tasks, the sensitivity (accuracy, precision) of CQA-AO strategy reached 0.940 (0.945, 0.948), 0.962 (0.937, 0.913), and 0.967 (0.962, 0.957) for brainstem, parotid and mandible, respectively. The unacceptable contour category analysis, the(FI,AccI,Fmicro,Fmacro)of CQA-AO strategy reached (0.901, 0.763, 0.862, 0.822), (0.855, 0.737, 0.837, 0.784), and (0.907, 0.762, 0.858, 0.821) for brainstem, parotid and mandible, respectively. After adaptive optimization correction, the DSC values of brainstem, parotid and mandible have been improved by 9.4%, 25.9%, and 13.5%, and Hausdorff distance values decreased by 62%, 70.6%, and 81.6%, respectively.Significance. The proposed CQA-AO strategy, which combines QA of contour and adaptive optimization correction for OARs contouring, demonstrated superior performance compare to conventional methods. This method can be implemented in the clinical contouring procedures and improve the efficiency of delineating and reviewing workflow.

Efficient quality assurance for isocentric stability in stereotactic body radiation therapy using machine learning

This study aims to predict isocentric stability for stereotactic body radiation therapy (SBRT) treatments using machine learning (ML), covers the challenges of manual assessment and computational time for quality assurance (QA), and supports medical physicists to enhance accuracy. The isocentric parameters for collimator (C), gantry (G), and table (T) tests were conducted with the RUBY phantom during QA using TrueBeam linac for SBRT. This analysis combined statistical features from the IsoCheck EPID software. Five ML models, including logistic regression (LR), decision tree (DT), random forest (RF), naive Bayes (NB), and support vector machines (SVM), were used to predict the outcome of the QA procedure. 247 Winston-Lutz (WL) tests were collected from 2020 to 2022. In our study, both DT and RF achieved the highest score on test accuracy (Acc. test) ranging from 93.5% to 99.4%, and area under curve (AUC) values from 90 to 100% on three modes (C, G, and T). The precision, recall, and F1 scores indicate the DT model consistently outperforms other ML models in predicting isocenter stability deviation in QA. The QA assessment using ML models can assist error prediction early to avoid potential harm during SBRT and ensure safe and effective patient treatments.

Contour subregion error detection methodology using deep learning auto-segmentation

BACKGROUND

Inaccurate manual organ delineation is one of the high-risk failure modes in radiation treatment. Numerous automated contour quality assurance (QA) systems have been developed to assess contour acceptability; however, manual inspection of flagged cases is a time-consuming and challenging process, and can lead to users overlooking the exact error location.

PURPOSE

Our aim is to develop and validate a contour QA system that can effectively detect and visualize subregional contour errors, both qualitatively and quantitatively.

METHODS/MATERIALS

A novel contour subregion error detection (CSED) system was developed using subregional surface distance discrepancies between manual and deep learning auto-segmentation (DLAS) contours. A validation study was conducted using a head and neck public dataset containing 339 cases and evaluated according to knowledge-based pass criteria derived from a clinical training dataset of 60 cases. A blind qualitative evaluation was conducted, comparing the results from the CSED system with manual labels. Subsequently, the CSED-flagged cases were re-examined by a radiation oncologist.

RESULTS

The CSED system could visualize the diverse types of subregional contour errors qualitatively and quantitatively. In the validation dataset, the CSED system resulted in true positive rates (TPR) of 0.814, 0.800, and 0.771; false positive rates (FPR) of 0.310, 0.267, and 0.298; and accuracies of 0.735, 0.759, and 0.730, for brainstem and left and right parotid contours, respectively. The CSED-assisted manual review caught 13 brainstem, 19 left parotid, and 21 right parotid contour errors missed by conventional human review. The TPR/FPR/accuracy of the CSED-assisted manual review improved to 0.836/0.253/0.784, 0.831/0.171/0.830, and 0.808/0.193/0.807 for each structure, respectively. Further, the time savings achieved through CSED-assisted review improved by 75%, with the time for review taking 24.81 ± 12.84, 26.75 ± 10.41, and 28.71 ± 13.72 s for each structure, respectively.

CONCLUSIONS

The CSED system enables qualitative and quantitative detection, localization, and visualization of manual segmentation subregional errors utilizing DLAS contours as references. The use of this system has been shown to help reduce the risk of high-risk failure modes resulting from inaccurate organ segmentation.

A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer

Background and Purpose

Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining.

Materials and Methods

Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics.

Results

The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum D 1 cm 3 . The bladder D 5cm 3 reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans.

Conclusions

Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning.

Clinical target volume delineation quality assurance for MRI-guided prostate radiotherapy using deep learning with uncertainty estimation

BACKGROUND AND PURPOSE

Previous studies on automatic delineation quality assurance (QA) have mostly focused on CT-based planning. As MRI-guided radiotherapy is increasingly utilized in prostate cancer treatment, there is a need for more research on MRI-specific automatic QA. This work proposes a clinical target volume (CTV) delineation QA framework based on deep learning (DL) for MRI-guided prostate radiotherapy.

MATERIALS AND METHODS

The proposed workflow utilized a 3D dropblock ResUnet++ (DB-ResUnet++) to generate multiple segmentation predictions via Monte Carlo dropout which were used to compute an average delineation and area of uncertainty. A logistic regression (LR) classifier was employed to classify the manual delineation as pass or discrepancy based on the spatial association between the manual delineation and the network's outputs. This approach was evaluated on a multicentre MRI-only prostate radiotherapy dataset and compared with our previously published QA framework based on AN-AG Unet.

RESULTS

The proposed framework achieved an area under the receiver operating curve (AUROC) of 0.92, a true positive rate (TPR) of 0.92 and a false positive rate of 0.09 with an average processing time per delineation of 1.3 min. Compared with our previous work using AN-AG Unet, this method generated fewer false positive detections at the same TPR with a much faster processing speed.

CONCLUSION

To the best of our knowledge, this is the first study to propose an automatic delineation QA tool using DL with uncertainty estimation for MRI-guided prostate radiotherapy, which can potentially be used for reviewing prostate CTV delineation in multicentre clinical trials.

Machine Learning-Based Quality Assurance for Automatic Segmentation of Head-and-Neck Organs-at-Risk in Radiotherapy

Purpose/Objective(s): With the development of deep learning, more convolutional neural networks (CNNs) are being introduced in automatic segmentation to reduce oncologists' labor requirement. However, it is still challenging for oncologists to spend considerable time evaluating the quality of the contours generated by the CNNs. Besides, all the evaluation criteria, such as Dice Similarity Coefficient (DSC), need a gold standard to assess the quality of the contours. To address these problems, we propose an automatic quality assurance (QA) method using isotropic and anisotropic methods to automatically analyze contour quality without a gold standard. Materials/Methods: We used 196 individuals with 18 different head-and-neck organs-at-risk. The overall process has the following 4 main steps. (1) Use CNN segmentation network to generate a series of contours, then use these contours as organ masks to erode and dilate to generate inner/outer shells for each 2D slice. (2) Thirty-eight radiomics features were extracted from these 2 shells, using the inner/outer shells' radiomics features ratios and DSCs as the input for 12 machine learning models. (3) Using the DSC threshold adaptively classified the passing/un-passing slices. (4) Through 2 different threshold analysis methods quantitatively evaluated the un-passing slices and obtained a series of location information of poor contours. Parts 1-3 were isotropic experiments, and part 4 was the anisotropic method. Result: From the isotropic experiments, almost all the predicted values were close to the labels. Through the anisotropic method, we obtained the contours' location information by assessing the thresholds of the peak-to-peak and area-to-area ratios. Conclusion: The proposed automatic segmentation QA method could predict the segmentation quality qualitatively. Moreover, the method can analyze the location information for un-passing slices.

Artificial intelligence can use physiological parameters to optimize treatment strategies and predict clinical deterioration of sepsis in ICU

Objective.Sepsis seriously threatens human life. Early identification of a patient's risk status and appropriate treatment can reduce septic shock risk and mortality. Our purpose is to design and validate an adjunctive therapy system based on deep reinforcement learning (DRL), which can provide treatment recommendations with providence and assess the patient's risk status and treatment options in the early stages.Approach.Data is from the Beth Israel Deaconess Medical Center. The raw data included 53 423 patients from MIMIC-III. Of these, 19 620 eligible samples were screened to form the final cohort. First, the patient's physiological parameters were fed into the DRL therapy strategy recommendation module (TSRM), which provides a forward-looking recommendation for treatment strategy. The recommended strategies were then fed into the reinforcement learning risk assessment module (RAM), which predicts the patient's risk status and treatment strategy from a long-term perspective. The DRL model designed in this paper assists in formulating treatment plans and evaluating treatment risks and patient status through continuous interaction with patient trajectory; this model therefore has the foresight that a supervising deep learning model does not.Main results.The experiment shows that, in the test set for the TSRM, mortality is the lowest when the treatment strategy that is actually implemented is the same as the AI-recommended strategy. Regarding the RAM, it can accurately grasp a patient's deterioration trend, and can reasonably assess a patient's risk status and treatment plans at an early stage. The assessment results of the model were matched with the actual clinical records.Significance.A DRL-based sepsis adjunctive therapy model is proposed. It can prospectively assist physicians in proposing treatment strategies, assess the patient's risk status and treatment methods early on, and detect deterioration trends in advance.

Artificial intelligence in radiotherapy

Radiotherapy is a discipline closely integrated with computer science. Artificial intelligence (AI) has developed rapidly over the past few years. With the explosive growth of medical big data, AI promises to revolutionize the field of radiotherapy through highly automated workflow, enhanced quality assurance, improved regional balances of expert experiences, and individualized treatment guided by multi-omics. In addition to independent researchers, the increasing number of large databases, biobanks, and open challenges significantly facilitated AI studies on radiation oncology. This article reviews the latest research, clinical applications, and challenges of AI in each part of radiotherapy including image processing, contouring, planning, quality assurance, motion management, and outcome prediction. By summarizing cutting-edge findings and challenges, we aim to inspire researchers to explore more future possibilities and accelerate the arrival of AI radiotherapy.