User-controlled pipelines for feature integration and head and neck radiation therapy outcome predictions

Ensemble learning approach for prediction of early complications after radiotherapy for head and neck cancer using CT and MRI radiomic features

There are different side effects in radiotherapy of head and neck cancer (HNC) including xerostomia. The present study utilizes the addition of [Formula: see text] and [Formula: see text]-weighted Magnetic Resonance (MR) radiomic image features to typical Computed Tomography (CT)-based features and radiation dose-based characteristics and incorporates the evaluation and validation of individual and ensemble classifiers for prediction of early-onset xerostomia in radiotherapy of HNC. A total of 80 patients diagnosed with HNC were evaluated prospectively. The dataset was divided into two subsets: 70% for training and validation and 30% for testing. Stratified random sampling was used to ensure that the proportion of xerostomia cases was consistent across both subsets. This approach preserved the class balance and ensured a representative distribution of demographic, dosimetric, and radiomic features in both sets. MR and CT imaging, dosimetric, and demographic features of patients were used as model input data. Bilateral parotid radiomic features were extracted from CT, [Formula: see text], and [Formula: see text] weighted MR images. Pearson statistical tests were used for selection of features and Random Tree (RT), Neural Network (NN), Linear Support Vector Machine (LSVM) and Bayesian Network (BN) classifiers were evaluated. To prevent overfitting and data leakage, preprocessing was conducted. All features were normalized using the z-score technique, with the mean and standard deviation calculated from the training set and then applied to the test set. For the training dataset, the Synthetic Minority Oversampling Technique (SMOTE) was used to balance the minority class. The results suggest the extracted features from [Formula: see text] weighted images have superior prediction ability compared to [Formula: see text] weighted acquisitions. The RT and BN models based on [Formula: see text] weighted images show better performance than those obtained with [Formula: see text] weighted images. [Formula: see text] weighted image-based analysis shows area under the curve (AUC) values for The RT and BN models of 0.90 and 0.84, respectively, while corresponding values obtained from [Formula: see text] weighted images are 0.79 and 0.78 for RT and BN models respectively. Combined [Formula: see text] weighted image-based models RT-BN, RT-LSVM-BN and RT-NN-LSVM-BN also show good performance having AUC values 0.97, 0.92, and 0.90, respectively. These results show that radiomic features from MR images obtained before radiotherapy can be used in addition to other metrics as personalized and unique biomarkers for prediction of early-onset xerostomia. Ensemble classifiers are more efficient than individual classifiers in prediction of early xerostomia.

Novel pre-spatial data fusion deep learning approach for multimodal volumetric outcome prediction models in radiotherapy

BACKGROUND

Given the recent increased emphasis on multimodal neural networks to solve complex modeling tasks, the problem of outcome prediction for a course of treatment can be framed as fundamentally multimodal in nature. A patient's response to treatment will vary based on their specific anatomy and the proposed treatment plan-these factors are spatial and closely related. However, additional factors may also have importance, such as non-spatial descriptive clinical characteristics, which can be structured as tabular data. It is critical to provide models with as comprehensive of a patient representation as possible, but inputs with differing data structures are incompatible in raw form; traditional models that consider these inputs require feature engineering prior to modeling. In neural networks, feature engineering can be organically integrated into the model itself, under one governing optimization, rather than performed prescriptively beforehand. However, the native incompatibility of different data structures must be addressed. Methods to reconcile structural incompatibilities in multimodal model inputs are called data fusion. We present a novel joint early pre-spatial (JEPS) fusion technique and demonstrate that differences in fusion approach can produce significant model performance differences even when the data is identical.

PURPOSE

To present a novel pre-spatial fusion technique for volumetric neural networks and demonstrate its impact on model performance for pretreatment prediction of overall survival (OS).

METHODS

From a retrospective cohort of 531 head and neck patients treated at our clinic, we prepared an OS dataset of 222 data-complete cases at a 2-year post-treatment time threshold. Each patient's data included CT imaging, dose array, approved structure set, and a tabular summary of the patient's demographics and survey data. To establish single-modality baselines, we fit both a Cox Proportional Hazards model (CPH) and a dense neural network on only the tabular data, then we trained a 3D convolutional neural network (CNN) on only the volume data. Then, we trained five competing architectures for fusion of both modalities: two early fusion models, a late fusion model, a traditional joint fusion model, and the novel JEPS, where clinical data is merged into training upstream of most convolution operations. We used standardized 10-fold cross validation to directly compare the performance of all models on identical train/test splits of patients, using area under the receiver-operator curve (AUC) as the primary performance metric. We used a two-tailed Student t-test to assess the statistical significance (p-value threshold 0.05) of any observed performance differences.

RESULTS

The JEPS design scored the highest, achieving a mean AUC of 0.779 ± 0.080. The late fusion model and clinical-only CPH model scored second and third highest with 0.746 ± 0.066 and 0.720 ± 0.091 mean AUC, respectively. The performance differences between these three models were not statistically significant. All other comparison models scored significantly worse than the top performing JEPS model.

CONCLUSION

For our OS evaluation, our JEPS fusion architecture achieves better integration of inputs and significantly improves predictive performance over most common multimodal approaches. The JEPS fusion technique is easily applied to any volumetric CNN.

Machine learning in image-based outcome prediction after radiotherapy: A review

The integration of machine learning (ML) with radiotherapy has emerged as a pivotal innovation in outcome prediction, bringing novel insights amid unique challenges. This review comprehensively examines the current scope of ML applications in various treatment contexts, focusing on treatment outcomes such as patient survival, disease recurrence, and treatment-induced toxicity. It emphasizes the ascending trajectory of research efforts and the prominence of survival analysis as a clinical priority. We analyze the use of several common medical imaging modalities in conjunction with clinical data, highlighting the advantages and complexities inherent in this approach. The research reflects a commitment to advancing patient-centered care, advocating for expanded research on abdominal and pancreatic cancers. While data collection, patient privacy, standardization, and interpretability present significant challenges, leveraging ML in radiotherapy holds remarkable promise for elevating precision medicine and improving patient care outcomes.

Deep reinforcement learning and convolutional autoencoders for anomaly detection of congenital inner ear malformations in clinical CT images

Detection of abnormalities within the inner ear is a challenging task even for experienced clinicians. In this study, we propose an automated method for automatic abnormality detection to provide support for the diagnosis and clinical management of various otological disorders. We propose a framework for inner ear abnormality detection based on deep reinforcement learning for landmark detection which is trained uniquely in normative data. In our approach, we derive two abnormality measurements: Dimage and Uimage. The first measurement, Dimage, is based on the variability of the predicted configuration of a well-defined set of landmarks in a subspace formed by the point distribution model of the location of those landmarks in normative data. We create this subspace using Procrustes shape alignment and Principal Component Analysis projection. The second measurement, Uimage, represents the degree of hesitation of the agents when approaching the final location of the landmarks and is based on the distribution of the predicted Q-values of the model for the last ten states. Finally, we unify these measurements in a combined anomaly measurement called Cimage. We compare our method's performance with a 3D convolutional autoencoder technique for abnormality detection using the patch-based mean squared error between the original and the generated image as a basis for classifying abnormal versus normal anatomies. We compare both approaches and show that our method, based on deep reinforcement learning, shows better detection performance for abnormal anatomies on both an artificial and a real clinical CT dataset of various inner ear malformations with an increase of 11.2% of the area under the ROC curve. Our method also shows more robustness against the heterogeneous quality of the images in our dataset.

Improved outcome models with denoising diffusion

PURPOSE

Radiotherapy outcome modelling often suffers from class imbalance in the modelled endpoints. One of the main options to address this issue is by introducing new synthetically generated datapoints, using generative models, such as Denoising Diffusion Probabilistic Models (DDPM). In this study, we implemented DDPM to improve performance of a tumor local control model, trained on imbalanced dataset, and compare this approach with other common techniques.

METHODS

A dataset of 535 NSCLC patients treated with SBRT (50 Gy/5 fractions) was used to train a deep learning outcome model for tumor local control prediction. The dataset included complete treatment planning data (planning CT images, 3D planning dose distribution and patient demographics) with sparsely distributed endpoints (6-7 % experiencing local failure). Consequently, we trained a novel conditional 3D DDPM model to generate synthetic treatment planning data. Synthetically generated treatment planning datapoints were used to supplement the real training dataset and the improvement in the model's performance was studied. Obtained results were also compared to other common techniques for class imbalanced training, such as Oversampling, Undersampling, Augmentation, Class Weights, SMOTE and ADASYN.

RESULTS

Synthetic DDPM-generated data were visually trustworthy, with Fréchet inception distance (FID) below 50. Extending the training dataset with the synthetic data improved the model's performance by more than 10%, while other techniques exhibited only about 4% improvement.

CONCLUSIONS

DDPM introduces a novel approach to class-imbalanced outcome modelling problems. The model generates realistic synthetic radiotherapy planning data, with a strong potential to increase performance and robustness of outcome models.

Prediction of toxicity outcomes following radiotherapy using deep learning-based models: A systematic review

PURPOSE

This study aims to perform a comprehensive systematic review of deep learning (DL) models in predicting RT-induced toxicity.

MATERIALS AND METHODS

A literature review was performed following the PRISMA guidelines. Systematic searches were performed in PubMed, Scopus, Cochrane and Embase databases from the earliest record up to September 2022. Related studies on deep learning models for radiotherapy toxicity prediction were selected based on predefined PICOS criteria.

RESULTS

Fourteen studies of radiotherapy-treated patients on different types of cancer [prostate (n=2), HNC (n=4), liver (n=2), lung (n=4), cervical (n=1), and oesophagus (n=1)] were eligible for inclusion in the systematic review. Information regarding patient characteristics and model development was summarized. Several approaches, such as ensemble learning, data augmentation, and transfer learning, that were utilized by selected studies were discussed.

CONCLUSION

Deep learning techniques are able to produce a consistent performance for toxicity prediction. Future research using large and diverse datasets and standardization of the study methodologies are required to improve the consistency of the research output.

Artificial intelligence-driven radiomics study in cancer: the role of feature engineering and modeling

Modern medicine is reliant on various medical imaging technologies for non-invasively observing patients' anatomy. However, the interpretation of medical images can be highly subjective and dependent on the expertise of clinicians. Moreover, some potentially useful quantitative information in medical images, especially that which is not visible to the naked eye, is often ignored during clinical practice. In contrast, radiomics performs high-throughput feature extraction from medical images, which enables quantitative analysis of medical images and prediction of various clinical endpoints. Studies have reported that radiomics exhibits promising performance in diagnosis and predicting treatment responses and prognosis, demonstrating its potential to be a non-invasive auxiliary tool for personalized medicine. However, radiomics remains in a developmental phase as numerous technical challenges have yet to be solved, especially in feature engineering and statistical modeling. In this review, we introduce the current utility of radiomics by summarizing research on its application in the diagnosis, prognosis, and prediction of treatment responses in patients with cancer. We focus on machine learning approaches, for feature extraction and selection during feature engineering and for imbalanced datasets and multi-modality fusion during statistical modeling. Furthermore, we introduce the stability, reproducibility, and interpretability of features, and the generalizability and interpretability of models. Finally, we offer possible solutions to current challenges in radiomics research.

Predicting Local Failure after Partial Prostate Re-Irradiation Using a Dosiomic-Based Machine Learning Model

The aim of this study is to predict local failure after partial prostate re-irradiation for the treatment of isolated locally recurrent prostate cancer by using a machine learning classifier based on radiomic features from pre-treatment computed tomography (CT), positron-emission tomography (PET) and biological effective dose distribution (BED) of the radiotherapy plan. The analysis was conducted on a monocentric dataset of 43 patients with evidence of isolated intraprostatic recurrence of prostate cancer after primary external beam radiotherapy. All patients received partial prostate re-irradiation delivered by volumetric modulated arc therapy. The gross tumor volume (GTV) of each patient was manually contoured from planning CT, choline-PET and dose maps. An ensemble machine learning pipeline including unbalanced data correction and feature selection was trained using the radiomic and dosiomic features as input for predicting occurrence of local failure. The model performance was assessed using sensitivity, specificity, accuracy and area under receiver operating characteristic curves of the score function in 10-fold cross validation repeated 100 times. Local failure was observed in 13 patients (30%), with a median time to recurrence of 36.7 months (range = 6.1-102.4 months). A four variables ensemble machine learning model resulted in accuracy of 0.62 and AUC 0.65. According to our results, a dosiomic machine learning classifier can predict local failure after partial prostate re-irradiation.

Bias and Class Imbalance in Oncologic Data-Towards Inclusive and Transferrable AI in Large Scale Oncology Data Sets

Recent technological developments have led to an increase in the size and types of data in the medical field derived from multiple platforms such as proteomic, genomic, imaging, and clinical data. Many machine learning models have been developed to support precision/personalized medicine initiatives such as computer-aided detection, diagnosis, prognosis, and treatment planning by using large-scale medical data. Bias and class imbalance represent two of the most pressing challenges for machine learning-based problems, particularly in medical (e.g., oncologic) data sets, due to the limitations in patient numbers, cost, privacy, and security of data sharing, and the complexity of generated data. Depending on the data set and the research question, the methods applied to address class imbalance problems can provide more effective, successful, and meaningful results. This review discusses the essential strategies for addressing and mitigating the class imbalance problems for different medical data types in the oncologic domain.

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

Radial Data Mining to Identify Density-Dose Interactions That Predict Distant Failure Following SABR

Purpose

Lower dose outside the planned treatment area in lung stereotactic radiotherapy has been linked to increased risk of distant metastasis (DM) possibly due to underdosage of microscopic disease (MDE). Independently, tumour density on pretreatment computed tomography (CT) has been linked to risk of MDE. No studies have investigated the interaction between imaging biomarkers and incidental dose. The interaction would showcase whether the impact of dose on outcome is dependent on imaging and, hence, if imaging could inform which patients require dose escalation outside the gross tumour volume (GTV). We propose an image-based data mining methodology to investigate density-dose interactions radially from the GTV to predict DM with no a priori assumption on location.

Methods

Dose and density were quantified in 1-mm annuli around the GTV for 199 patients with early-stage lung cancer treated with 60 Gy in 5 fractions. Each annulus was summarised by three density and three dose parameters. For parameter combinations, Cox regressions were performed including a dose-density interaction in independent annuli. Heatmaps were created that described improvement in DM prediction due to the interaction. Regions of significant improvement were identified and studied in overall outcome models.

Results

Dose-density interactions were identified that significantly improved prediction for over 50% of bootstrap resamples. Dose and density parameters were not significant when the interaction was omitted. Tumour density variance and high peritumour density were associated with DM for patients with more cold spots (less than 30-Gy EQD2) and non-uniform dose about 3 cm outside of the GTV. Associations identified were independent of the mean GTV dose.

Conclusions

Patients with high tumour variance and peritumour density have increased risk of DM if there is a low and non-uniform dose outside the GTV. The dose regions are independent of tumour dose, suggesting that incidental dose may play an important role in controlling occult disease. Understanding such interactions is key to identifying patients who will benefit from dose-escalation. The methodology presented allowed spatial dose-density interactions to be studied at the exploratory stage for the first time. This could accelerate the clinical implementation of imaging biomarkers by demonstrating the impact of incidental dose for tumours of varying characteristics in routine data.

Deep Learning for Radiotherapy Outcome Prediction Using Dose Data - A Review

Artificial intelligence, and in particular deep learning using convolutional neural networks, has been used extensively for image classification and segmentation, including on medical images for diagnosis and prognosis prediction. Use in radiotherapy prognostic modelling is still limited, however, especially as applied to toxicity and tumour response prediction from radiation dose distributions. We review and summarise studies that applied deep learning to radiotherapy dose data, in particular studies that utilised full three-dimensional dose distributions. Ten papers have reported on deep learning models for outcome prediction utilising spatial dose information, whereas four studies used reduced dimensionality (dose volume histogram) information for prediction. Many of these studies suffer from the same issues that plagued early normal tissue complication probability modelling, including small, single-institutional patient cohorts, lack of external validation, poor data and model reporting, use of late toxicity data without taking time-to-event into account, and nearly exclusive focus on clinician-reported complications. They demonstrate, however, how radiation dose, imaging and clinical data may be technically integrated in convolutional neural networks-based models; and some studies explore how deep learning may help better understand spatial variation in radiosensitivity. In general, there are a number of issues specific to the intersection of radiotherapy outcome modelling and deep learning, for example translation of model developments into treatment plan optimisation, which will require further combined effort from the radiation oncology and artificial intelligence communities.

Exploratory ensemble interpretable model for predicting local failure in head and neck cancer: the additive benefit of CT and intra-treatment cone-beam computed tomography features

BACKGROUND

Local failure (LF) following chemoradiation (CRT) for head and neck cancer is associated with poor overall survival. If machine learning techniques could stratify patients at risk of treatment failure based on baseline and intra-treatment imaging, such a model could facilitate response-adapted approaches to escalate, de-escalate, or switch therapy.

METHODS

A 1:2 retrospective case control cohort of patients treated at a single institution with definitive radiotherapy for head and neck cancer who failed locally, in-field at a primary or nodal structure were included. Radiomic features were extracted from baseline CT and CBCT scans at fractions 1 and 21 (delta) of radiotherapy with PyRadiomics and were selected for by: reproducibility (intra-class correlation coefficients ≥0.95), redundancy [maximum relevance and minimum redundancy (mRMR)], and informativeness [recursive feature elimination (RFE)]. Separate models predicting LF of primaries or nodes were created using the explainable boosting machine (EBM) classifier with 5-fold cross-validation for (I) clinical only, (II) radiomic only (CT1 and delta features), and (III) fused models (clinical + radiomic). Twenty-five iterations were performed, and predicted scores were averaged with a parallel ensemble design. Receiver operating characteristic curves were compared between models with paired-samples t-tests.

RESULTS

The fused ensemble model for primaries (using clinical, CT1, and delta features) achieved an AUC of 0.871 with a sensitivity of 78.3% and specificity of 90.9% at the maximum Youden J statistic. The fused ensemble model trended towards improvement when compared to the clinical only ensemble model (AUC =0.788, P=0.134) but reached significance when compared to the radiomic ensemble model (AUC =0.770, P=0.017). The fused ensemble model for nodes achieved an AUC of 0.910 with a sensitivity of 100.0% and specificity of 68.0%, which also trended towards improvement when compared to the clinical model (AUC =0.865, P=0.080).

CONCLUSIONS

The fused ensemble EBM model achieved high discriminatory ability at predicting LF for head and neck cancer in independent primary and nodal structures. Although an additive benefit of delta radiomics over clinical factors could not be proven, the results trended towards improvement with the fused ensemble model, which are promising and worthy of prospective investigation in a larger cohort.

Combining computed tomography and biologically effective dose in radiomics and deep learning improves prediction of tumor response to robotic lung stereotactic body radiation therapy

PURPOSE

The aim of this study is to improve the performance of machine learning (ML) models in predicting response of non-small cell lung cancer (NSCLC) to stereotactic body radiation therapy (SBRT) by integrating image features from pre-treatment computed tomography (CT) with features from the biologically effective dose (BED) distribution.

MATERIALS AND METHODS

Image features, consisting of crafted radiomic features or machine-learned features extracted using a convolutional neural network, were calculated from pre-treatment CT data and from dose distributions converted into BED for 80 NSCLC lesions over 76 patients treated with robotic guided SBRT. ML models using different combinations of features were trained to predict complete or partial response according to response criteria in solid tumors, including radiomics CT (Rad_CT ), radiomics CT and BED (Rad_CT,BED ), deep learning (DL) CT (DL_CT ), and DL CT and BED (DL_CT,BED ). Training of ML included feature selection by neighborhood component analysis followed by ensemble ML using robust boosting. A model was considered as acceptable when the sum of average sensitivity and specificity on test data in repeated cross validations was at least 1.5.

RESULTS

Complete or partial response occurred in 58 out of 80 lesions. The best models to predict the tumor response were those using BED variables, achieving significantly better area under curve (AUC) and accuracy than those using only features from CT, including a Rad_CT,BED model using three radiomic features from BED, which scored an accuracy of 0.799 (95% confidence intervals (0.75-0.85)) and AUC of 0.773 (0.688-0.846), and a DL_CT,BED model also using three variables with an accuracy of 0.798 (0.649-0.829) and AUC of 0.812 (0.755-0.867).

CONCLUSION

According to our results, the inclusion of BED features improves the response prediction of ML models for lung cancer patients undergoing SBRT, regardless of the use of radiomic or DL features.

Artificial Intelligence and the Medical Physicist: Welcome to the Machine

Artificial intelligence (AI) is a branch of computer science dedicated to giving machines or computers the ability to perform human-like cognitive functions, such as learning, problem-solving, and decision making. Since it is showing superior performance than well-trained human beings in many areas, such as image classification, object detection, speech recognition, and decision-making, AI is expected to change profoundly every area of science, including healthcare and the clinical application of physics to healthcare, referred to as medical physics. As a result, the Italian Association of Medical Physics (AIFM) has created the “AI for Medical Physics” (AI4MP) group with the aims of coordinating the efforts, facilitating the communication, and sharing of the knowledge on AI of the medical physicists (MPs) in Italy. The purpose of this review is to summarize the main applications of AI in medical physics, describe the skills of the MPs in research and clinical applications of AI, and define the major challenges of AI in healthcare.