Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality

Radiomics on spatial-temporal manifolds via Fokker-Planck dynamics

BACKGROUND

Delta radiomics is a high-throughput computational technique used to describe quantitative changes in serial, time-series imaging by considering the relative change in radiomic features of images extracted at two distinct time points. Recent work has demonstrated a lack of prognostic signal of radiomic features extracted using this technique. We hypothesize that this lack of signal is due to the fundamental assumptions made when extracting features via delta radiomics, and that other methods should be investigated.

PURPOSE

The purpose of this work was to show a proof-of-concept of a new radiomics paradigm for sparse, time-series imaging data, where features are extracted from a spatial-temporal manifold modeling the time evolution between images, and to assess the prognostic value on patients with oropharyngeal cancer (OPC).

METHODS

To accomplish this, we developed an algorithm to mathematically describe the relationship between two images acquired at timet = 0 $t = 0$ andt > 0 $t > 0$ . These images serve as boundary conditions of a partial differential equation describing the transition from one image to the other. To solve this equation, we propagate the position and momentum of each voxel according to Fokker-Planck dynamics (i.e., a technique common in statistical mechanics). This transformation is driven by an underlying potential force uniquely determined by the equilibrium image. The solution generates a spatial-temporal manifold (3 spatial dimensions + time) from which we define dynamic radiomic features. First, our approach was numerically verified by stochastically sampling dynamic Gaussian processes of monotonically decreasing noise. The transformation from high to low noise was compared between our Fokker-Planck estimation and simulated ground-truth. To demonstrate feasibility and clinical impact, we applied our approach to 18F-FDG-PET images to estimate early metabolic response of patients (n = 57) undergoing definitive (chemo)radiation for OPC. Images were acquired pre-treatment and 2-weeks intra-treatment (after 20 Gy). Dynamic radiomic features capturing changes in texture and morphology were then extracted. Patients were partitioned into two groups based on similar dynamic radiomic feature expression via k-means clustering and compared by Kaplan-Meier analyses with log-rank tests (p < 0.05). These results were compared to conventional delta radiomics to test the added value of our approach.

RESULTS

Numerical results confirmed our technique can recover image noise characteristics given sparse input data as boundary conditions. Our technique was able to model tumor shrinkage and metabolic response. While no delta radiomics features proved prognostic, Kaplan-Meier analyses identified nine significant dynamic radiomic features. The most significant feature was Gray-Level-Size-Zone-Matrix gray-level variance (p = 0.011), which demonstrated prognostic improvement over its corresponding delta radiomic feature (p = 0.722).

CONCLUSIONS

We developed, verified, and demonstrated the prognostic value of a novel, physics-based radiomics approach over conventional delta radiomics via data assimilation of quantitative imaging and differential equations.

Dose-Incorporated Deep Ensemble Learning for Improving Brain Metastasis Stereotactic Radiosurgery Outcome Prediction

PURPOSE

To develop a novel deep ensemble learning model for accurate prediction of brain metastasis (BM) local control outcomes after stereotactic radiosurgery (SRS).

METHODS AND MATERIALS

A total of 114 brain metastases (BMs) from 82 patients were evaluated, including 26 BMs that developed biopsy-confirmed local failure post-SRS. The SRS spatial dose distribution (Dmap) of each BM was registered to the planning contrast-enhanced T1 (T1-CE) magnetic resonance imaging (MRI). Axial slices of the Dmap, T1-CE, and planning target volume (PTV) segmentation (PTVseg) intersecting the BM center were extracted within a fixed field of view determined by the 60% isodose volume in Dmap. A spherical projection was implemented to transform planar image content onto a spherical surface using multiple projection centers, and the resultant T1-CE/Dmap/PTVseg projections were stacked as a 3-channel variable. Four Visual Geometry Group (VGG-19) deep encoders were used in an ensemble design, with each submodel using a different spherical projection formula as input for BM outcome prediction. In each submodel, clinical features after positional encoding were fused with VGG-19 deep features to generate logit results. The ensemble's outcome was synthesized from the 4 submodel results via logistic regression. In total, 10 model versions with random validation sample assignments were trained to study model robustness. Performance was compared with (1) a single VGG-19 encoder, (2) an ensemble with a T1-CE MRI as the sole image input after projections, and (3) an ensemble with the same image input design without clinical feature inclusion.

RESULTS

The ensemble model achieved an excellent area under the receiver operating characteristic curve (AUCROC: 0.89 ± 0.02) with high sensitivity (0.82 ± 0.05), specificity (0.84 ± 0.11), and accuracy (0.84 ± 0.08) results. This outperformed the MRI-only VGG-19 encoder (sensitivity: 0.35 ± 0.01, AUCROC: 0.64 ± 0.08), the MRI-only deep ensemble (sensitivity: 0.60 ± 0.09, AUCROC: 0.68 ± 0.06), and the 3-channel ensemble without clinical feature fusion (sensitivity: 0.78 ± 0.08, AUCROC: 0.84 ± 0.03).

CONCLUSIONS

Facilitated by the spherical image projection method, a deep ensemble model incorporating Dmap and clinical variables demonstrated excellent performance in predicting BM post-SRS local failure. Our novel approach could improve other radiation therapy outcome models and warrants further evaluation.

Development of a multi-feature-combined model: proof-of-concept with application to local failure prediction of post-SBRT or surgery early-stage NSCLC patients

Objective

To develop a Multi-Feature-Combined (MFC) model for proof-of-concept in predicting local failure (LR) in NSCLC patients after surgery or SBRT using pre-treatment CT images. This MFC model combines handcrafted radiomic features, deep radiomic features, and patient demographic information in an integrated machine learning workflow.

Methods

The MFC model comprised three key steps. (1) Extraction of 92 handcrafted radiomic features from the GTV segmented on pre-treatment CT images. (2) Extraction of 512 deep radiomic features from pre-trained U-Net encoder. (3) The extracted handcrafted radiomic features, deep radiomic features, along with 4 patient demographic information (i.e., gender, age, tumor volume, and Charlson comorbidity index), were concatenated as a multi-dimensional input to the classifiers for LR prediction. Two NSCLC patient cohorts from our institution were investigated: (1) the surgery cohort includes 83 patients with segmentectomy or wedge resection (7 LR), and (2) the SBRT cohort includes 84 patients with lung SBRT (9 LR). The MFC model was developed and evaluated independently for both cohorts, and was subsequently compared against the prediction models based on only handcrafted radiomic features (R models), patient demographic information (PI models), and deep learning modeling (DL models). ROC with AUC was adopted to evaluate model performance with leave-one-out cross-validation (LOOCV) and 100-fold Monte Carlo random validation (MCRV). The t-test was performed to identify the statistically significant differences.

Results

In LOOCV, the AUC range (surgery/SBRT) of the MFC model was 0.858-0.895/0.868-0.913, which was higher than the three other models: 0.356-0.480/0.322-0.650 for PI models, 0.559-0.618/0.639-0.682 for R models, and 0.809/0.843 for DL models. In 100-fold MCRV, the MFC model again showed the highest AUC results (surgery/SBRT): 0.742-0.825/0.888-0.920, which were significantly higher than PI models: 0.464-0.564/0.538-0.628, R models: 0.557-0.652/0.551-0.732, and DL models: 0.702/0.791.

Conclusion

We successfully developed an MFC model that combines feature information from multiple sources for proof-of-concept prediction of LR in patients with surgical and SBRT early-stage NSCLC. Initial results suggested that incorporating pre-treatment patient information from multiple sources improves the ability to predict the risk of local failure.