An investigation into the risk of population bias in deep learning autocontouring

Guidance on selecting and evaluating AI auto-segmentation systems in clinical radiotherapy: insights from a six-vendor analysis

Artificial Intelligence (AI) based auto-segmentation has demonstrated numerous benefits to clinical radiotherapy workflows. However, the rapidly changing regulatory, research, and market environment presents challenges around selecting and evaluating the most suitable solution. To support the clinical adoption of AI auto-segmentation systems, Selection Criteria recommendations were developed to enable a holistic evaluation of vendors, considering not only raw performance but associated risks uniquely related to the clinical deployment of AI. In-house experience and key bodies of work on ethics, standards, and best practices for AI in Radiation Oncology were reviewed to inform selection criteria and evaluation strategies. A retrospective analysis using the criteria was performed across six vendors, including a quantitative assessment using five metrics (Dice, Hausdorff Distance, Average Surface Distance, Surface Dice, Added Path Length) across 20 head and neck, 20 thoracic, and 19 male pelvis patients for AI models as of March 2023. A total of 47 selection criteria were identified across seven categories. A retrospective analysis showed that overall no vendor performed exceedingly well, with systematically poor performance in Data Security & Responsibility, Vendor Support Tools, and Transparency & Ethics. In terms of raw performance, vendors varied widely from excellent to poor. As new regulations come into force and the scope of AI auto-segmentation systems adapt to clinical needs, continued interest in ensuring safe, fair, and transparent AI will persist. The selection and evaluation framework provided herein aims to promote user confidence by exploring the breadth of clinically relevant factors to support informed decision-making.

Incorporating patient-specific prior clinical knowledge to improve clinical target volume auto-segmentation generalisability for online adaptive radiotherapy of rectal cancer: A multicenter validation

BACKGROUND & PURPOSE

Deep learning (DL) based auto-segmentation has shown to be beneficial for online adaptive radiotherapy (OART). However, auto-segmentation of clinical target volumes (CTV) is complex, as clinical interpretations are crucial in their definition. The resulting variation between clinicians and institutes hampers the generalizability of DL networks. In OART the CTV is delineated during treatment preparation which makes the clinician intent explicitly available during treatment. We studied whether multicenter generalisability improves when using this prior clinical knowledge, the pre-treatment delineation, as a patient-specific prior for DL models for online auto-segmentation of the mesorectal CTV.

MATERIAL & METHODS

We included intermediate risk or locally advanced rectal cancer patients from three centers. Patient-specific weight maps were created by combining the patient-specific CTV delineation on the pre-treatment scan with population-based variation of likely inter-fraction mesorectal CTV deformations. We trained two models to auto-segment the mesorectal CTV on in-house data, one with (MRI + prior) and one without (MRI-only) priors. Both models were applied to two external datasets. An external baseline model was trained without priors from scratch for one external center. Performance was evaluated on the DSC, surface Dice, 95HD and MSD.

RESULTS

For both external centers, the MRI + prior model outperformed the MRI-only model significantly on the segmentation metrics (p-values < 0.01). There was no significant difference between the external baseline model and the MRI + prior model.

CONCLUSION

Adding patient-specific weight maps makes the CTV segmentation model more robust to institutional preferences. Performance was comparable to a model trained locally from scratch. This makes this approach suitable for generalization to multiple centers.

Automated segmentation in planning-CT for breast cancer radiotherapy: A review of recent advances

Postoperative radiotherapy (RT) has been shown to effectively reduce disease recurrence and mortality in breast cancer (BC) treatment. A critical step in the planning workflow is the accurate delineation of clinical target volumes (CTV) and organs-at-risk (OAR). This literature review evaluates recent advancements in deep-learning (DL) and atlas-based auto-contouring techniques for CTVs and OARs in BC planning-CT images for RT. It examines their performance regarding geometrical and dosimetric accuracy, inter-observer variability, and time efficiency. Our findings indicate that both DL- and atlas-based methods generally show comparable performance across OARs and CTVs, with DL methods slightly outperforming in consistency and accuracy. Auto-segmentation of breast and most OARs achieved robust results in both segmentation quality and dosimetric planning. However, lymph node levels (LNLs) presented the greatest challenge in auto-segmentation with significant impact on dosimetric planning. The translation of these findings into clinical practice is limited by the geometric performance metrics and the lack of dose evaluation studies. Additionally, auto-contouring algorithms showed diverse structure sets, while training datasets varied in size, origin, patient positioning and imaging protocols, affecting model sensitivity. Guideline inconsistencies and varying definitions of ground truth led to substantial variability, suggesting a need for a reliable consensus training dataset. Finally, our review highlights the popularity of the U-Net architecture. In conclusion, while automated contouring has proven efficient for many OARs and the breast-CTV, further improvements are necessary in LNL delineation, dosimetric analysis, and consensus building.

Cardiac substructure delineation in radiation therapy - A state-of-the-art review

Delineation of cardiac substructures is crucial for a better understanding of radiation-related cardiotoxicities and to facilitate accurate and precise cardiac dose calculation for developing and applying risk models. This review examines recent advancements in cardiac substructure delineation in the radiation therapy (RT) context, aiming to provide a comprehensive overview of the current level of knowledge, challenges and future directions in this evolving field. Imaging used for RT planning presents challenges in reliably visualising cardiac anatomy. Although cardiac atlases and contouring guidelines aid in standardisation and reduction of variability, significant uncertainties remain in defining cardiac anatomy. Coupled with the inherent complexity of the heart, this necessitates auto-contouring for consistent large-scale data analysis and improved efficiency in prospective applications. Auto-contouring models, developed primarily for breast and lung cancer RT, have demonstrated performance comparable to manual contouring, marking a significant milestone in the evolution of cardiac delineation practices. Nevertheless, several key concerns require further investigation. There is an unmet need for expanding cardiac auto-contouring models to encompass a broader range of cancer sites. A shift in focus is needed from ensuring accuracy to enhancing the robustness and accessibility of auto-contouring models. Addressing these challenges is paramount for the integration of cardiac substructure delineation and associated risk models into routine clinical practice, thereby improving the safety of RT for future cancer patients.

Artificial intelligence uncertainty quantification in radiotherapy applications - A scoping review

BACKGROUND/PURPOSE

The use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions.

METHODS

We followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics.

RESULTS

We identified 56 articles published from 2015 to 2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50 %), followed by image-synthesis (13 %), and multiple applications simultaneously (11 %). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32 %). Imaging data was used in 91 % of studies, while only 13 % incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60 %), with Monte Carlo dropout being the most commonly implemented UQ method (32 %) followed by ensembling (16 %). 55 % of studies did not share code or datasets.

CONCLUSION

Our review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, we identified a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.

Is full-automation in radiotherapy treatment planning ready for take off?

Radiotherapy treatment planning is undergoing a transformation with the increasing integration of automation. This transition draws parallels with the aviation industry, which has a long-standing history of addressing challenges and opportunities introduced by automated systems. Both fields witness a shift from manual operations to systems capable of operating independently, raising questions about the risks and evolving role of humans within automated workflows. In response to this shift, a working group assembled during the ESTRO Physics Workshop 2023, reflected on parallels to draw lessons for radiotherapy. A taxonomy is proposed, leveraging insights from aviation, that outlines the observed levels of automation within the context of radiotherapy and their corresponding implications for human involvement. Among the common identified risks associated with automation integration are complacency, overreliance, attention tunneling, data overload, a lack of transparency and training. These risks require mitigation strategies. Such strategies include ensuring role complementarity, introducing checklists and safety requirements for human-automation interaction and using automation for cognitive unload and workflow management. Focusing on already automated processes, such as dose calculation and auto-contouring as examples, we have translated lessons learned from aviation. It remains crucial to strike a balance between automation and human involvement. While automation offers the potential for increased efficiency and accuracy, it must be complemented by human oversight, expertise, and critical decision-making. The irreplaceable value of human judgment remains, particularly in complex clinical situations. Learning from aviation, we identify a need for human factors engineering research in radiation oncology and a continued requirement for proactive incident learning.

Rapid Selection of Patients Suitable for Deep Inspiration Breath-Hold Using an Automatic Delineating System and RapidPlan Model in Patients With Left Breast Cancer Undergoing Adjuvant Radiation Therapy With IMRT

PURPOSE

This study aimed to determine whether radiation therapy plans created using an automatic delineating system and a RapidPlan (RP) module could rapidly and accurately predict heart doses and benefit from deep inspiratory breath-hold (DIBH) in patients with left breast cancer.

METHODS AND MATERIALS

One hundred thirty-six clinically approved free breathing (FB) plans for patients with left breast cancer were included, defined as manual delineation-manual plan (MD-MP). A total of 104 of 136 plans were selected for RP model training. A total of 32 of 136 patients were automatically delineated by software, after which the RP generated plans, defined as automatic delineation-RapidPlan (AD-RP). In addition, 40 patients who used DIBH were included to analyze differences in heart benefits from DIBH.

RESULTS

Two RP models were established for post-breast-conserving surgery (BCS) and post-modified radical mastectomy. There were no significant differences in most of the dosimetric parameters between the MD-MP and AD-RP. The heart doses of the 2 plans were strongly correlated in patients after BCS (0.80 ≤ r ≤ 0.88, P < .05) and moderately correlated in patients after postmodified radical mastectomy (0.46 ≤ r ≤ 0.58, P <.05). The RP model predicted the mean heart dose (MHD) within ± 59.67 cGy and ± 63.32 cGy for patients who underwent the 2 surgeries described above. The heart benefits from DIBH were significantly greater in patients with FB-MHD ≥ 4 Gy than in those with FB-MHD < 4 Gy.

CONCLUSIONS

The combined automatic delineation RP model allows for the rapid and accurate prediction of heart dose under FB in patients with left breast cancer. FB-MHD ≥ 4 Gy can be used as a dose threshold to select patients suitable for DIBH.

Experience of Implementing Deep Learning-Based Automatic Contouring in Breast Radiation Therapy Planning: Insights From Over 2000 Cases

PURPOSE

This study evaluated the impact and clinical utility of an auto-contouring system for radiation therapy treatments.

METHODS AND MATERIALS

The auto-contouring system was implemented in 2019. We evaluated data from 2428 patients who underwent adjuvant breast radiation therapy before and after the system's introduction. We collected the treatment's finalized contours, which were reviewed and revised by a multidisciplinary team. After implementation, the treatment contours underwent a finalization process that involved manual review and adjustment of the initial auto-contours. For the preimplementation group (n = 369), auto-contours were generated retrospectively. We compared the auto-contours and final contours using the Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95).

RESULTS

We analyzed 22,215 structures from final and corresponding auto-contours. The final contours were generally larger, encompassing more slices in the superior or inferior directions. Among organs at risk (OAR), the heart, esophagus, spinal cord, and contralateral breast demonstrated significantly increased DSC and decreased HD95 postimplementation (all P < .05), except for the lungs, which presented inaccurate segmentation. Among target volumes, CTVn_L2, L3, L4, and the internal mammary node showed increased DSC and decreased HD95 postimplementation (all P < .05), although the increase was less pronounced than the OAR outcomes. The analysis also covered factors contributing to significant differences, pattern identification, and outlier detection.

CONCLUSIONS

In our study, the adoption of an auto-contouring system was associated with an increased reliance on automated settings, underscoring its utility and the potential risk of automation bias. Given these findings, we underscore the importance of considering the integration of stringent risk assessments and quality management strategies as a precautionary measure for the optimal use of such systems.

Artificial Intelligence Uncertainty Quantification in Radiotherapy Applications - A Scoping Review

Background/purpose

The use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions.

Methods

We followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics.

Results

We identified 56 articles published from 2015-2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50%), followed by image-synthesis (13%), and multiple applications simultaneously (11%). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32%). Imaging data was used in 91% of studies, while only 13% incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60%), with Monte Carlo dropout being the most commonly implemented UQ method (32%) followed by ensembling (16%). 55% of studies did not share code or datasets.

Conclusion

Our review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, there was a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.

Clinical evaluation on automatic segmentation results of convolutional neural networks in rectal cancer radiotherapy

Purpose

Image segmentation can be time-consuming and lacks consistency between different oncologists, which is essential in conformal radiotherapy techniques. We aimed to evaluate automatic delineation results generated by convolutional neural networks (CNNs) from geometry and dosimetry perspectives and explore the reliability of these segmentation tools in rectal cancer.

Methods

Forty-seven rectal cancer cases treated from February 2018 to April 2019 were randomly collected retrospectively in our cancer center. The oncologists delineated regions of interest (ROIs) on planning CT images as the ground truth, including clinical target volume (CTV), bladder, small intestine, and femoral heads. The corresponding automatic segmentation results were generated by DeepLabv3+ and ResUNet, and we also used Atlas-Based Autosegmentation (ABAS) software for comparison. The geometry evaluation was carried out using the volumetric Dice similarity coefficient (DSC) and surface DSC, and critical dose parameters were assessed based on replanning optimized by clinically approved or automatically generated CTVs and organs at risk (OARs), i.e., the Planref and Plantest. Pearson test was used to explore the correlation between geometric metrics and dose parameters.

Results

In geometric evaluation, DeepLabv3+ performed better in DCS metrics for the CTV (volumetric DSC, mean = 0.96, P< 0.01; surface DSC, mean = 0.78, P< 0.01) and small intestine (volumetric DSC, mean = 0.91, P< 0.01; surface DSC, mean = 0.62, P< 0.01), ResUNet had advantages in volumetric DSC of the bladder (mean = 0.97, P< 0.05). For critical dose parameters analysis between Planref and Plantest, there was a significant difference for target volumes (P< 0.01), and no significant difference was found for the ResUNet-generated small intestine (P > 0.05). For the correlation test, a negative correlation was found between DSC metrics (volumetric, surface DSC) and dosimetric parameters (δD95, δD95, HI, CI) for target volumes (P< 0.05), and no significant correlation was found for most tests of OARs (P > 0.05).

Conclusions

CNNs show remarkable repeatability and time-saving in automatic segmentation, and their accuracy also has a certain potential in clinical practice. Meanwhile, clinical aspects, such as dose distribution, may need to be considered when comparing the performance of auto-segmentation methods.