Optimizing efficiency and safety in external beam radiotherapy using automated plan check (APC) tool and six sigma methodology

Analysis of human errors in the operation of various treatment planning systems over a 10-year period

The present study aimed to summarize and report data on errors related to treatment planning, which were collected by medical physicists. The following analyses were performed based on the 10-year error report data: (1) listing of high-risk errors that occurred and (2) the relationship between the number of treatments and error rates, (3) usefulness of the Automated Plan Checking System (APCS) with the Eclipse Scripting Application Programming Interface and (4) the relationship between human factors and error rates. Differences in error rates were observed before and after the use of APCS. APCS reduced the error rate by ~1% for high-risk errors and 3% for low-risk errors. The number of treatments was negatively correlated with error rates. Therefore, we examined the relationship between the workload of medical physicists and error occurrence and revealed that a very large workload may contribute to overlooking errors. Meanwhile, an increase in the number of medical physicists may lead to the detection of more errors. The number of errors was correlated with the number of physicians with less clinical experience; the error rates were higher when there were more physicians with less experience. This is likely due to the lack of training among clinically inexperienced physicians. An environment to provide adequate training is important, as inexperience in clinical practice can easily and directly lead to the occurrence of errors. In any environment, the need for additional plan checkers is an essential factor for eliminating errors.

Applying the six-sigma methodology to determine the limits of quality control (QC) tests for a specific linear accelerator

PURPOSE

We aimed to show the framework of the six-sigma methodology (SSM) that can be used to determine the limits of QC tests for the linear accelerator (Linac). Limits for QC tests are individually determined using the SSM.

METHODS AND MATERIALS

The SSM is based on the define-measure-analyze-improve-control (DMAIC) stages to improve the process. In the "define" stage, the limits of QC tests were determined. In the "measure" stage, a retrospective collection of daily QC data using a Machine Performance Check platform was performed from January 2020 to December 2022. In the "analyze" stage, the process of determining the limits was proposed using statistical analyses and process capability indices. In the "improve" stage, the capability index was used to calculate the action limits. The tolerance limit was established using the larger one of the control limits in the individual control chart (I-chart). In the "control" stage, daily QC data were collected prospectively from January 2023 to May 2023 to monitor the effect of action limits and tolerance limits.

RESULTS

A total of 798 sets of QC data including beam, isocenter, collimation, couch, and gantry tests were collected and analyzed. The Collimation Rotation offset test had the min-Cp, min-Cpk, min-Pp, and min-Ppk at 2.53, 1.99, 1.59, and 1.25, respectively. The Couch Rtn test had the max-Cp, max-Cpk, max-Pp, and max-Ppk at 31.5, 29.9, 23.4, and 22.2, respectively. There are three QC tests with higher action limits than the original tolerance. Some data on the I-chart of the beam output change, isocenter KV offset, and jaw X1 exceeded the lower tolerance and action limit, which indicated that a system deviation occurred and reminded the physicist to take action to improve the process.

CONCLUSIONS

The SSM is an excellent framework to use in determining the limits of QC tests. The process capability index is an important parameter that provides quantitative information on determining the limits of QC tests.

Enhancing patient safety in radiotherapy: Implementation of a customized electronic checklist for radiation therapists

Introduction

The radiotherapy workflow involves the collaboration of multiple professionals and the execution of several steps to results in an effective treatment. In this study, we described the clinical implementation of an electronic checklist, developed to standardize the process of the chart review prior to the first treatment fraction by the radiation therapists (RTTs).

Materials and Methods

A customized electronic checklist was developed based on the recommendations of American Association of Physicists in Medicine (AAPM) Task Groups 275 and 315 and integrated into the Record and Verify System (RVS). The checklist consisted of 16 items requiring binary (yes/no) responses, with mandatory completion and review by RTTs prior to treatment. The utility of the checklist and its impact on workflow were assessed by analysing checklist reports, and by soliciting feedback to RTTs through an anonymized survey.

Results

During the first trial phase, from June to November 2023, 285 checklists were completed with a 98% compilation rate and 94.4% review rate. Forty errors were detected, mainly due to missing signed treatment plans and absence of Beam's Eye View documentation. Ninety percent of detected errors were fixed before the treatment start. In 4 cases, the problem could not be fixed before the first fraction, resulting in a suboptimal first treatment. The feedback survey showed that RTTs described the checklist as useful, with minimal impact on workload, and supported its implementation.

Discussion

The introduction of a customized electronic checklist improved the detection and correction of errors, thereby enhancing patient safety. The positive response from RTTs and the minimal impact on workflow underscore the value of the checklist as standard practice in radiotherapy departments.

Process failure mode and effects analysis for external beam radiotherapy: Introducing a literature-based template and a novel action priority

PURPOSE

The first aim of the study was to create a general template for analyzing potential failures in external beam radiotherapy, EBRT, using the process failure mode and effects analysis (PFMEA). The second aim was to modify the action priority (AP), a novel prioritization method originally introduced by the Automotive Industry Action Group (AIAG), to work with different severity, occurrence, and detection rating systems used in radiation oncology.

METHODS AND MATERIALS

The AIAG PFMEA approach was employed in combination with an extensive literature survey to develop the EBRT-PFMEA template. Subsets of high-risk failure modes found through the literature survey were added to the template where applicable. Our modified AP for radiation oncology (RO AP) was defined using a weighted sum of severity, occurrence, and detectability. Then, Monte Carlo simulations were conducted to compare the original AIAG AP, the RO AP, and the risk priority number (RPN). The results of the simulations were used to determine the number of additional corrective actions per failure mode and to parametrize the RO AP to our department's rating system.

RESULTS

An EBRT-PFMEA template comprising 75 high-risk failure modes could be compiled. The AIAG AP required 1.7 additional corrective actions per failure mode, while the RO AP ranged from 1.3 to 3.5, and the RPN required 3.6. The RO AP could be parametrized so that it suited our rating system and evaluated severity, occurrence, and detection ratings equally to the AIAG AP.

CONCLUSIONS

An adjustable EBRT-PFMEA template is provided which can be used as a practical starting point for creating institution-specific templates. Moreover, the RO AP introduces transparent action levels that can be adapted to any rating system.

Machine learning and lean six sigma for targeted patient-specific quality assurance of volumetric modulated arc therapy plans

Background and purpose

Radiotherapy plans with excessive complexity exhibit higher uncertainties and worse patient-specific quality assurance (PSQA) results, while the workload of measurement-based PSQA can impact the efficiency of the radiotherapy workflow. Machine Learning (ML) and Lean Six Sigma, a process optimization method, were implemented to adopt a targeted PSQA approach, aiming to reduce workload, risk of failures, and monitor complexity.

Materials and methods

Lean Six Sigma was applied using DMAIC (define, measure, analyze, improve, and control) steps. Ten complexity metrics were computed for 69,811 volumetric modulated arc therapy (VMAT) arcs from 28,612 plans delivered in our Institute (2013-2021). Outlier complexities were defined as >95th-percentile of the historical distributions, stratified by treatment. An ML model was trained to predict the gamma passing rate (GPR-3 %/1mm) of an arc given its complexity. A decision support system was developed to monitor the complexity and expected GPR. Plans at risk of PSQA failure, either extremely complex or with average GPR <90 %, were identified. The tool's impact was assessed after nine months of clinical use.

Results

Among 1722 VMAT plans monitored prospectively, 29 (1.7 %) were found at risk of failure. Planners reacted by performing PSQA measurement and re-optimizing the plan. Occurrences of outlier complexities remained stable within 5 %. The expected GPR increased from a median of 97.4 % to 98.2 % (Mann-Whitney p < 0.05) due to plan re-optimization.

Conclusions

ML and Lean Six Sigma have been implemented in clinical practice enabling a targeted measurement-based PSQA approach for plans at risk of failure to improve overall quality and patient safety.

Technical Note: Improving the workflow in a carbon ion therapy center with custom software for enhanced patient care

Carbon-ion radiation therapy (CIRT) is an up-and-coming modality for cancer treatment. Implementation of CIRT requires collaboration among specialists like radiation oncologists, medical physicists, and other healthcare professionals. Effective communication among team members is necessary for the success of CIRT. However, the current workflows involving data management, treatment planning, scheduling, and quality assurance (QA) can be susceptible to errors, leading to delays and decreased efficiency. With the aim of addressing these challenges, a team of medical physicists developed an in-house workflow management software using FileMaker Pro. This tool has streamlined the workflow and improved the efficiency and quality of patient care.

Automated contouring, treatment planning, and quality assurance for VMAT craniospinal irradiation (VMAT-CSI)

Purpose

Create a comprehensive automated solution for pediatric and adult VMAT-CSI including contouring, planning, and plan check to reduce planning time and improve plan quality.

Methods

Seventy-seven previously treated CSI patients (age, 2-67 years) were used for creation of an auto-contouring model to segment 25 organs at risk (OARs). The auto-contoured OARs were evaluated using the Dice Similarity Coefficient (DSC), 95% Hausdorff Distance (HD95), and a qualitative ranking by one physician and one physicist (scale: 1-acceptable, 2-minor edits, 3-major edits). The auto-planning script was developed using the Varian Eclipse Scripting API and tested with 20 patients previously treated with either low-dose VMAT-CSI (12 Gy) or high-dose VMAT-CSI (36 Gy + 18 Gy boost). Clinically relevant metrics, planning time, and blinded physician review were used to evaluate significance of differences between the auto and manual plans. Finally, the plan preparation for treatment and plan check processes were automated to improve efficiency and safety of VMAT-CSI.

Results

The auto-contours achieved an average DSC of 0.71 ± 0.15, HD95 of 4.81 ± 4.68, and reviewers' ranking of 1.22 ± 0.39, indicating close to "acceptable-as-is" contours. Compared to the manual CSI plans, the auto-plans for both dose regimens achieved statistically significant reductions in body V50% and Dmean for parotids, submandibular, and thyroid glands. The variance in the dosimetric parameters decreased for the auto-plans as compared to the manual plans indicating better plan consistency. From the blinded review, the auto-plans were marked as equivalent or superior to the manual-plans 88.3% of the time. The required time for the auto-contouring and planning was consistently between 1-2 hours compared to an estimated 5-6 hours for manual contouring and planning.

Conclusions

Reductions in contouring and planning time without sacrificing plan quality were obtained using the developed auto-planning process. The auto-planning scripts and documentation will be made freely available to other institutions and clinics.

Unified risk analysis in radiation therapy

PURPOSE

The increasing complexity of new treatment methods as well as the Information Technology (IT) infrastructure within radiotherapy require new methods for risk analysis. This work presents a methodology on how to model the treatment process of radiotherapy in different levels. This subdivision makes it possible to perform workflow-specific risk analysis and to assess the impact of IT risks on the overall treatment workflow.

METHODS

A Unified Modeling Language (UML) activity diagram is used to model the workflows. The subdivision of the workflows into different levels is done with the help of swim lanes. The model created in this way is exported in an xml-compatible format and stored in a database with the help of a Python program.

RESULTS

Based on an existing risk analysis, the workflows CT Appointment, Glioblastoma Multiforme, and Deep Inspiration Breath Hold (DIBH) were modeled in detail. Part of the analysis are automatically generated workflow-specific risk matrices including risks of medical devices incorporated into a specific workflow. In addition, SQL queries allow to quickly retrieve e.g., the details of the medical device network installed in a department.

CONCLUSION

Activity diagrams of UML can be used to model workflows in radiotherapy. Through this, a connection between the different levels of the entire workflow can be established and workflow-specific risk analysis is possible.

Implementation of a knowledge-based decision support system for treatment plan auditing through automation

BACKGROUND

Independent auditing is a necessary component of a comprehensive quality assurance (QA) program and can also be utilized for continuous quality improvement (QI) in various radiotherapy processes. Two senior physicists at our institution have been performing a time intensive manual audit of cross-campus treatment plans annually, with the aim of further standardizing our planning procedures, updating policies and guidelines, and providing training opportunities of all staff members.

PURPOSE

A knowledge-based automated anomaly-detection algorithm to provide decision support and strengthen our manual retrospective plan auditing process was developed. This standardized and improved the efficiency of the assessment of our external beam radiotherapy (EBRT) treatment planning across all eight campuses of our institution.

METHODS

A total of 843 external beam radiotherapy plans for 721 lung patients from January 2020 to March 2021 were automatically acquired from our clinical treatment planning and management systems. From each plan, 44 parameters were automatically extracted and pre-processed. A knowledge-based anomaly detection algorithm, namely, "isolation forest" (iForest), was then applied to the plan dataset. An anomaly score was determined for each plan using recursive partitioning mechanism. Top 20 plans ranked with the highest anomaly scores for each treatment technique (2D/3D/IMRT/VMAT/SBRT) including auto-populated parameters were used to guide the manual auditing process and validated by two plan auditors.

RESULTS

The two auditors verified that 75.6% plans with the highest iForest anomaly scores have similar concerning qualities that may lead to actionable recommendations for our planning procedures and staff training materials. The time to audit a chart was approximately 20.8 min on average when done manually and 14.0 min when done with the iForest guidance. Approximately 6.8 min were saved per chart with the iForest method. For our typical internal audit review of 250 charts annually, the total time savings are approximately 30 hr per year.

CONCLUSION

iForest effectively detects anomalous plans and strengthens our cross-campus manual plan auditing procedure by adding decision support and further improve standardization. Due to the use of automation, this method was efficient and will be used to establish a standard plan auditing procedure, which could occur more frequently.

External validation of a prediction model for timely implementation of innovations in radiotherapy

BACKGROUND AND PURPOSE

The aim of this study was to externally validate a model that predicts timely innovation implementation, which can support radiotherapy professionals to be more successful in innovation implementation.

MATERIALS AND METHODS

A multivariate prediction model was built based on the TRIPOD (Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis) criteria for a type 4 study (1). The previously built internally validated model had an AUC of 0.82, and was now validated using a completely new multicentre dataset. Innovation projects that took place between 2017-2019 were included in this study. Semi-structured interviews were performed to retrieve the prognostic variables of the previously built model. Projects were categorized according to the size of the project; the success of the project and thepresence of pre-defined success factors were analysed.

RESULTS

Of the 80 included innovation projects (32.5% technological, 35% organisational and 32.5% treatment innovations), 55% were successfully implemented within the planned timeframe. Comparing the outcome predictions with the observed outcomes of all innovations resulted in an AUC of the external validation of the prediction model of 0.72 (0.60-0.84, 95% CI). Factors related to successful implementation included in the model are sufficient and competent employees, desirability and feasibility, clear goals and processes and the complexity of a project.

CONCLUSION

For the first time, a prediction model focusing on the timely implementation of innovations has been successfully built and externally validated. This model can now be widely used to enable more successful innovation in radiotherapy.

Automated Plan Checking Software Demonstrates Continuous and Sustained Improvements in Safety and Quality: A 3-year Longitudinal Analysis

PURPOSE

This study aimed to perform a longitudinal analysis of the performance of our automated plan checking software by retrospectively evaluating the number of errors identified in plans delivered to patients in 3, month-long, data collection periods between 2017 and 2020.

METHODS AND MATERIALS

Eleven automated checks were retrospectively run on 1169 external beam radiation therapy treatment plans identified as meeting the following criteria: planning target volume-based multifield photon plans receiving a status of treatment approved in March 2017, March 2018, or March 2020. The number of passes (true positives) and flags were recorded. Flags were subcategorized into false negatives, false negatives due to naming conventions, and true negatives. In addition, 2 × 2 contingency tables using a 2-tailed Fisher's exact test were used to determine whether there were nonrandom associations between the output of the automated plan checking software and whether the check was manual or automated at the original time of treatment approval.

RESULTS

A statistically significant decrease in flags between the pre- and postautomation data sets was observed for 4 contour-based checks, namely adjacent structures overlap, empty structures and missing slices, overlap between body and couch, and laterality, as well as a check that determined whether the plan's global maximum dose was within the planning target volume. A review of the origins of false negatives was fed back into the design of the checks to improve the reliability of the system and help avoid warning fatigue.

CONCLUSIONS

Periodic and longitudinal review of the performance of automated software was essential for monitoring and understanding its impact on error rates, as well as for optimization of the tool to adapt to regular changes of clinical practice. The automated plan checking software has demonstrated continuous contributions to the safe and effective delivery of external beam radiation therapy to our patient population, an impact that extends beyond its initial implementation and deployment.

Toward automation of initial chart check for photon/electron EBRT: the clinical implementation of new AAPM task group reports and automation techniques

Purpose

The recently published AAPM TG‐275 and the public review version of TG‐315 list new recommendations for comprehensive and minimum physics initial chart checks, respectively. This article addresses the potential development and benefit of initial chart check automation when these recommendations are implemented for clinical photon/electron EBRT.

Methods

Eight board‐certified physicists with 2–20 years of clinical experience performed initial chart checks using checklists from TG‐275 and TG‐315. Manual check times were estimated for three types of plans (IMRT/VMAT, 3D, and 2D) and for prostate, whole pelvis, lung, breast, head and neck, and brain cancers. An expert development team of three physicists re‐evaluated the automation feasibility of TG‐275 checklist based on their experience of developing and implementing the in‐house and the commercial automation tools in our institution. Three levels of initial chart check automation were simulated: (1) Auto_UMMS_tool (which consists of in‐house program and commercially available software); (2) Auto_TG275 (with full and partial automation as indicated in TG‐275); and (3) Auto_UMMS_exp (with full and partial automation as determined by our experts’ re‐evaluation).

Results

With no automation of initial chart checks, the ranges of manual check times were 29–56 min (full TG‐315 list) and 102–163 min (full TG‐275 list), which varied significantly with physicists but varied little at different tumor sites. The 69 of 71 checks which were considered as “not fully automated” in TG‐275 were re‐evaluated with more automation feasibility. Compared to no automation, the higher levels of automation yielded a great reduction in both manual check times (by 44%–98%) and potentially residual detectable errors (by 15–85%).

Conclusion

The initial chart check automation greatly improves the practicality and efficiency of implementing the new TG recommendations. Revisiting the TG reports with new technology/practice updates may help develop and utilize more automation clinically.

A Probability-Based Investigation on the Setup Robustness of Pencil-beam Proton Radiation Therapy for Skull-Base Meningioma

Introduction

The intracranial skull-base meningioma is in proximity to multiple critical organs and heterogeneous tissues. Steep dose gradients often result from avoiding critical organs in proton treatment plans. Dose uncertainties arising from setup errors under image-guided radiation therapy are worthy of evaluation.

Patients and Methods

Fourteen patients with skull-base meningioma were retrospectively identified and planned with proton pencil beam scanning (PBS) single-field uniform dose (SFUD) and multifield optimization (MFO) techniques. The setup uncertainties were assigned a probability model on the basis of prior published data. The impact on the dose distribution from nominal 1-mm and large, less probable setup errors, as well as the cumulative effect, was analyzed. The robustness of SFUD and MFO planning techniques in these scenarios was discussed.

Results

The target coverage was reduced and the plan dose hot spot increased by all setup uncertainty scenarios regardless of the planning techniques. For 1 mm nominal shifts, the deviations in clinical target volume (CTV) coverage D99% was -11 ± 52 cGy and -45 ± 147 cGy for SFUD and MFO plans. The setup uncertainties affected the organ at risk (OAR) dose both positively and negatively. The statistical average of the setup uncertainties had <100 cGy impact on the plan qualities for all patients. The cumulative deviations in CTV D95% were 1 ± 34 cGy and -7 ± 18 cGy for SFUD and MFO plans.

Conclusion

It is important to understand the impact of setup uncertainties on skull-base meningioma, as the tumor target has complex shape and is in proximity to multiple critical organs. Our work evaluated the setup uncertainty based on its probability distribution and evaluated the dosimetric consequences. In general, the SFUD plans demonstrated more robustness than the MFO plans in target coverages and brainstem dose. The probability-weighted overall effect on the dose distribution is small compared to the dosimetric shift during single fraction.

Graph-based risk assessment and error detection in radiation therapy

PURPOSE

The objective of this study was to formalize and automate quality assurance (QA) in radiation oncology. Quality assurance in radiation oncology entails a multistep verification of complex, personalized radiation plans to treat cancer involving an interdisciplinary team and high technology, multivendor software and hardware systems. We addressed the pretreatment physics chart review (TPCR) using methods from graph theory and constraint programming to study the effect of dependencies between variables and automatically identify logical inconsistencies and how they propagate.

MATERIALS AND METHODS

We used a modular approach to decompose the TPCR process into tractable units comprising subprocesses, modules and variables. Modules represented the main software entities comprised in the radiation treatment planning workflow and subprocesses grouped the checks to be performed by functionality. Module-associated variables served as inputs to the subprocesses. Relationships between variables were modeled by means of a directed graph. The detection of errors, in the form of inconsistencies, was formalized as a constraint satisfaction problem whereby checks were encoded as logical formulae. The sequence in which subprocesses were visited was described in an activity diagram.

RESULTS

The comprehensive model for the TPCR process comprised 5 modules, 19 subprocesses and 346 variables, 225 of which were distinct. Modules included "Treatment Planning System" and "Record and Verify System." Subprocesses included "Dose Prescription," "Documents," "CT Integrity," "Anatomical Contours," "Beam Configuration," "Dose Calculation," "3D Dose Distribution Quality," and "Treatment Approval." Variable inconsistencies, and their source and propagation were determined by checking for constraint violation and through graph traversal. Impact scores, obtained through graph traversal, combined with severity scores associated with an inconsistency, allowed risk assessment.

CONCLUSIONS

Directed graphs combined with constraint programming hold promise for formalizing complex QA processes in radiation oncology, performing risk assessment and automating the TPCR process. Though complex, the process is tractable.

Automation of DVH Constraint Checks and Physics Quality Control Review Improves Patient Safety in Radiotherapy

This study investigates whether patient safety can be enhanced by the implementation of an automated electronic checklist (PlanCheck) for physics quality control review (QCR) of radiotherapy photon plans. PlanCheck evaluates both technical aspects and DVH constraints. Three hundred and thirty-one consecutively approved radiotherapy plans previously reviewed with manual QCR were retrospectively checked with PlanCheck. Four hundred and thirty-three (3.4%) of the 12783 automated technical checks executed in the 331 plans yielded an error. All errors were scored using the severity rating from the American Association of Physicists in Medicine TG-100 report. Nineteen of these errors (4%) either could have affected or affected target dose (severity 5+) implicating a maximum dose difference to the target or a critical organ at risk of 0.5% to 10% and 3 errors could have resulted in stereotactic brain treatments being delivered to the wrong location (severity 10). Forty-seven breast cancer plans were retrospectively subjected to automated DVH check, 10 undocumented dose constraint violations were found. PlanCheck has been shown to reduce errors in manually reviewed radiotherapy plans and thus to enhance patient safety.

Combining automatic plan integrity check (APIC) with standard plan document and checklist method to reduce errors in treatment planning

Purpose/objectives

To report our experience of combining three approaches of an automatic plan integrity check (APIC), a standard plan documentation, and checklist methods to minimize errors in the treatment planning process.

Materials/methods

We developed APIC program and standardized plan documentation via scripting in the treatment planning system, with an enforce function of APIC usage. We used a checklist method to check for communication errors in patient charts (referred to as chart errors). Any errors in the plans and charts (referred to as the planning errors) discovered during the initial chart check by the therapists were reported to our institutional Workflow Enhancement (WE) system. Clinical Implementation of these three methods is a progressive process while the APIC was the major progress among the three methods. Thus, we chose to compared the total number of planning errors before (including data from 2013 to 2014) and after (including data from 2015 to 2018) APIC implementation. We assigned the severity of these errors into five categories: serious (S), near miss with safety net (NM), clinical interruption (CLI), minor impediment (MI), and bookkeeping (BK). The Mann–Whitney U test was used for statistical analysis.

Results

A total of 253 planning error forms, containing 272 errors, were submitted during the study period, representing an error rate of 3.8%, 3.1%, 2.1%, 0.8%, 1.9% and 1.3% of total number of plans in these years respectively. A marked reduction of planning error rate in the S and NM categories was statistically significant (P < 0.01): from 0.6% before APIC to 0.1% after APIC. The error rate for all categories was also significantly reduced (P < 0.01), from 3.4% before APIC and 1.5% per plan after APIC.

Conclusion

With three combined methods, we reduced both the number and the severity of errors significantly in the process of treatment planning.