An image regression motion prediction technique for MRI-guided radiotherapy evaluated in single-plane cine imaging

Modeling of artificial intelligence-based respiratory motion prediction in MRI-guided radiotherapy: a review

The advancement of precision radiotherapy techniques, such as volumetric modulated arc therapy (VMAT), stereotactic body radiotherapy (SBRT), and particle therapy, highlights the importance of radiotherapy in the treatment of cancer, while also posing challenges for respiratory motion management in thoracic and abdominal tumors. MRI-guided radiotherapy (MRIgRT) stands out as state-of-art real-time respiratory motion management approach owing to the non-ionizing radiation nature and superior soft-tissue contrast characteristic of MR imaging. In clinical practice, MR imaging often operates at a frequency of 4 Hz, resulting in approximately a 300 ms system latency of MRIgRT. This system latency decreases the accuracy of respiratory motion management in MRIgRT. Artificial intelligence (AI)-based respiratory motion prediction has recently emerged as a promising solution to address the system latency issues in MRIgRT, particularly for advanced contour prediction and volumetric prediction. However, implementing AI-based respiratory motion prediction faces several challenges including the collection of training datasets, the selection of prediction methods, and the formulation of complex contour and volumetric prediction problems. This review presents modeling approaches of AI-based respiratory motion prediction in MRIgRT, and provides recommendations for achieving consistent and generalizable results in this field.

Volumetric MRI with sparse sampling for MR-guided 3D motion tracking via sparse prior-augmented implicit neural representation learning

BACKGROUND

Volumetric reconstruction of magnetic resonance imaging (MRI) from sparse samples is desirable for 3D motion tracking and promises to improve magnetic resonance (MR)-guided radiation treatment precision. Data-driven sparse MRI reconstruction, however, requires large-scale training datasets for prior learning, which is time-consuming and challenging to acquire in clinical settings.

PURPOSE

To investigate volumetric reconstruction of MRI from sparse samples of two orthogonal slices aided by sparse priors of two static 3D MRI through implicit neural representation (NeRP) learning, in support of 3D motion tracking during MR-guided radiotherapy.

METHODS

A multi-layer perceptron network was trained to parameterize the NeRP model of a patient-specific MRI dataset, where the network takes 4D data coordinates of voxel locations and motion states as inputs and outputs corresponding voxel intensities. By first training the network to learn the NeRP of two static 3D MRI with different breathing motion states, prior information of patient breathing motion was embedded into network weights through optimization. The prior information was then augmented from two motion states to 31 motion states by querying the optimized network at interpolated and extrapolated motion state coordinates. Starting from the prior-augmented NeRP model as an initialization point, we further trained the network to fit sparse samples of two orthogonal MRI slices and the final volumetric reconstruction was obtained by querying the trained network at 3D spatial locations. We evaluated the proposed method using 5-min volumetric MRI time series with 340 ms temporal resolution for seven abdominal patients with hepatocellular carcinoma, acquired using golden-angle radial MRI sequence and reconstructed through retrospective sorting. Two volumetric MRI with inhale and exhale states respectively were selected from the first 30 s of the time series for prior embedding and augmentation. The remaining 4.5-min time series was used for volumetric reconstruction evaluation, where we retrospectively subsampled each MRI to two orthogonal slices and compared model-reconstructed images to ground truth images in terms of image quality and the capability of supporting 3D target motion tracking.

RESULTS

Across the seven patients evaluated, the peak signal-to-noise-ratio between model-reconstructed and ground truth MR images was 38.02 ± 2.60 dB and the structure similarity index measure was 0.98 ± 0.01. Throughout the 4.5-min time period, gross tumor volume (GTV) motion estimated by deforming a reference state MRI to model-reconstructed and ground truth MRI showed good consistency. The 95-percentile Hausdorff distance between GTV contours was 2.41 ± 0.77 mm, which is less than the voxel dimension. The mean GTV centroid position difference between ground truth and model estimation was less than 1 mm in all three orthogonal directions.

CONCLUSION

A prior-augmented NeRP model has been developed to reconstruct volumetric MRI from sparse samples of orthogonal cine slices. Only one exhale and one inhale 3D MRI were needed to train the model to learn prior information of patient breathing motion for sparse image reconstruction. The proposed model has the potential of supporting 3D motion tracking during MR-guided radiotherapy for improved treatment precision and promises a major simplification of the workflow by eliminating the need for large-scale training datasets.

Experimental comparison of linear regression and LSTM motion prediction models for MLC-tracking on an MRI-linac

BACKGROUND

Magnetic resonance imaging (MRI)-guided radiotherapy with multileaf collimator (MLC)-tracking is a promising technique for intra-fractional motion management, achieving high dose conformality without prolonging treatment times. To improve beam-target alignment, the geometric error due to system latency should be reduced by using temporal prediction.

PURPOSE

To experimentally compare linear regression (LR) and long-short-term memory (LSTM) motion prediction models for MLC-tracking on an MRI-linac using multiple patient-derived traces with different complexities.

METHODS

Experiments were performed on a prototype 1.0 T MRI-linac capable of MLC-tracking. A motion phantom was programmed to move a target in superior-inferior (SI) direction according to eight lung cancer patient respiratory motion traces. Target centroid positions were localized from sagittal 2D cine MRIs acquired at 4 Hz using a template matching algorithm. The centroid positions were input to one of four motion prediction models. We used (1) a LSTM network which had been optimized in a previous study on patient data from another cohort (offline LSTM). We also used (2) the same LSTM model as a starting point for continuous re-optimization of its weights during the experiment based on recent motion (offline+online LSTM). Furthermore, we implemented (3) a continuously updated LR model, which was solely based on recent motion (online LR). Finally, we used (4) the last available target centroid without any changes as a baseline (no-predictor). The predictions of the models were used to shift the MLC aperture in real-time. An electronic portal imaging device (EPID) was used to visualize the target and MLC aperture during the experiments. Based on the EPID frames, the root-mean-square error (RMSE) between the target and the MLC aperture positions was used to assess the performance of the different motion predictors. Each combination of motion trace and prediction model was repeated twice to test stability, for a total of 64 experiments.

RESULTS

The end-to-end latency of the system was measured to be (389 ± 15) ms and was successfully mitigated by both LR and LSTM models. The offline+online LSTM was found to outperform the other models for all investigated motion traces. It obtained a median RMSE over all traces of (2.8 ± 1.3) mm, compared to the (3.2 ± 1.9) mm of the offline LSTM, the (3.3 ± 1.4) mm of the online LR and the (4.4 ± 2.4) mm when using the no-predictor. According to statistical tests, differences were significant (p-value <0.05) among all models in a pair-wise comparison, but for the offline LSTM and online LR pair. The offline+online LSTM was found to be more reproducible than the offline LSTM and the online LR with a maximum deviation in RMSE between two measurements of 10%.

CONCLUSIONS

This study represents the first experimental comparison of different prediction models for MRI-guided MLC-tracking using several patient-derived respiratory motion traces. We have shown that among the investigated models, continuously re-optimized LSTM networks are the most promising to account for the end-to-end system latency in MRI-guided radiotherapy with MLC-tracking.

Evaluation of real-time tumor contour prediction using LSTM networks for MR-guided radiotherapy

BACKGROUND AND PURPOSE

Magnetic resonance imaging guided radiotherapy (MRgRT) with deformable multileaf collimator (MLC) tracking would allow to tackle both rigid displacement and tumor deformation without prolonging treatment. However, the system latency must be accounted for by predicting future tumor contours in real-time. We compared the performance of three artificial intelligence (AI) algorithms based on long short-term memory (LSTM) modules for the prediction of 2D-contours 500ms into the future.

MATERIALS AND METHODS

Models were trained (52 patients, 3.1h of motion), validated (18 patients, 0.6h) and tested (18 patients, 1.1h) with cine MRs from patients treated at one institution. Additionally, we used three patients (2.9h) treated at another institution as second testing set. We implemented 1) a classical LSTM network (LSTM-shift) predicting tumor centroid positions in superior-inferior and anterior-posterior direction which are used to shift the last observed tumor contour. The LSTM-shift model was optimized both in an offline and online fashion. We also implemented 2) a convolutional LSTM model (ConvLSTM) to directly predict future tumor contours and 3) a convolutional LSTM combined with spatial transformer layers (ConvLSTM-STL) to predict displacement fields used to warp the last tumor contour.

RESULTS

The online LSTM-shift model was found to perform slightly better than the offline LSTM-shift and significantly better than the ConvLSTM and ConvLSTM-STL. It achieved a 50% Hausdorff distance of 1.2mm and 1.0mm for the two testing sets, respectively. Larger motion ranges were found to lead to more substantial performance differences across the models.

CONCLUSION

LSTM networks predicting future centroids and shifting the last tumor contour are the most suitable for tumor contour prediction. The obtained accuracy would allow to reduce residual tracking errors during MRgRT with deformable MLC-tracking.

Real time volumetric MRI for 3D motion tracking via geometry-informed deep learning

PURPOSE

To develop a geometry-informed deep learning framework for volumetric MRI with sub-second acquisition time in support of 3D motion tracking, which is highly desirable for improved radiotherapy precision but hindered by the long image acquisition time.

METHODS

A 2D-3D deep learning network with an explicitly defined geometry module that embeds geometric priors of the k-space encoding pattern was investigated, where a 2D generation network first augmented the sparsely sampled image dataset by generating new 2D representations of the underlying 3D subject. A geometry module then unfolded the 2D representations to the volumetric space. Finally, a 3D refinement network took the unfolded 3D data and outputted high-resolution volumetric images. Patient-specific models were trained for seven abdominal patients to reconstruct volumetric MRI from both orthogonal cine slices and sparse radial samples. To evaluate the robustness of the proposed method to longitudinal patient anatomy and position changes, we tested the trained model on separate datasets acquired more than one month later and evaluated 3D target motion tracking accuracy using the model-reconstructed images by deforming a reference MRI with gross tumor volume (GTV) contours to a 5-min time series of both ground truth and model-reconstructed volumetric images with a temporal resolution of 340 ms.

RESULTS

Across the seven patients evaluated, the median distances between model-predicted and ground truth GTV centroids in the superior-inferior direction were 0.4 ± 0.3 mm and 0.5 ± 0.4 mm for cine and radial acquisitions, respectively. The 95-percentile Hausdorff distances between model-predicted and ground truth GTV contours were 4.7 ± 1.1 mm and 3.2 ± 1.5 mm for cine and radial acquisitions, which are of the same scale as cross-plane image resolution.

CONCLUSION

Incorporating geometric priors into deep learning model enables volumetric imaging with high spatial and temporal resolution, which is particularly valuable for 3D motion tracking and has the potential of greatly improving MRI-guided radiotherapy precision.

3D Kinect Camera Scheme with Time-Series Deep-Learning Algorithms for Classification and Prediction of Lung Tumor Motility

This paper proposes a time-series deep-learning 3D Kinect camera scheme to classify the respiratory phases with a lung tumor and predict the lung tumor displacement. Specifically, the proposed scheme is driven by two time-series deep-learning algorithmic models: the respiratory-phase classification model and the regression-based prediction model. To assess the performance of the proposed scheme, the classification and prediction models were tested with four categories of datasets: patient-based datasets with regular and irregular breathing patterns; and pseudopatient-based datasets with regular and irregular breathing patterns. In this study, 'pseudopatients' refer to a dynamic thorax phantom with a lung tumor programmed with varying breathing patterns and breaths per minute. The total accuracy of the respiratory-phase classification model was 100%, 100%, 100%, and 92.44% for the four dataset categories, with a corresponding mean squared error (MSE), mean absolute error (MAE), and coefficient of determination (R²) of 1.2-1.6%, 0.65-0.8%, and 0.97-0.98, respectively. The results demonstrate that the time-series deep-learning classification and regression-based prediction models can classify the respiratory phases and predict the lung tumor displacement with high accuracy. Essentially, the novelty of this research lies in the use of a low-cost 3D Kinect camera with time-series deep-learning algorithms in the medical field to efficiently classify the respiratory phase and predict the lung tumor displacement.

MRI-guided cardiac-induced target motion tracking for atrial fibrillation cardiac radioablation

BACKGROUND AND PURPOSE

Atrial fibrillation (AF) cardiac radioablation (CR) challenges radiotherapy tracking: multiple small targets close to organs-at-risk undergo rapid differential cardiac contraction and respiratory motion. MR-guidance offers a real-time target tracking solution. This work develops and investigates MRI-guided tracking of AF CR targets with cardiac-induced motion.

MATERIALS AND METHODS

A direct tracking method (Tracking_direct) and two indirect tracking methods leveraging population-based surrogacy relationships with the left atria (Tracking_{indirect_LA}) or other target (Tracking_{indirect_target}) were developed. Tracking performance was evaluated using transverse ECG-gated breathhold MRI images from 15 healthy and 10 AF participants. Geometric and volumetric tracking errors were calculated, defined as the difference between the ground-truth and tracked target centroids and volumes respectively. Transverse, breath-hold, noncardiac-gated cine images were acquired at 4 Hz in 5 healthy and 5 AF participants to qualitatively characterize tracking performance on images more comparable to MRILinac acquisitions.

RESULTS

The average 3D geometric tracking errors for Tracking_direct, Tracking_{indirect_LA} and Tracking_{indirect_target} respectively were 1.7 ± 1.2 mm, 1.6 ± 1.1 mm and 1.9 ± 1.3 mm in healthy participants and 1.7 ± 1.3 mm, 1.5 ± 1.0 mm and 1.7 ± 1.2 mm in AF participants. For Tracking_direct, 88% of analyzed images had 3D geometric tracking errors <3 mm and the average volume tracking error was 1.7 ± 1.3 cc. For Tracking_direct on non-cardiac-gated cine images, tracked targets overlapped organsat-risk or completely missed the target area on 2.2% and 0.08% of the images respectively.

CONCLUSION

The feasibility of non-invasive MRI-guided tracking of cardiac-induced AF CR target motion was demonstrated for the first time, showing potential for improving AF CR treatment efficacy.

Volumetric prediction of breathing and slow drifting motion in the abdomen using radial MRI and multi-temporal resolution modeling

Abdominal organ motions introduce geometric uncertainties to radiotherapy. This study investigates a multi-temporal resolution 3D motion prediction scheme that accounts for both breathing and slow drifting motion in the abdomen in support of MRI-guided radiotherapy. Ten-minute MRI scans were acquired for 8 patients using a volumetric golden-angle stack-of-stars sequence. The first five-minutes was used for patient-specific motion modeling. Fast breathing motion was modeled from high temporal resolution radial k-space samples, which served as a navigator signal to sort k-space data into different bins for high spatial resolution reconstruction of breathing motion states. Slow drifting motion was modeled from a lower temporal resolution image time series which was reconstructed by sequentially combining a large number of breathing-corrected k-space samples. Principal components analysis (PCA) was performed on deformation fields between different motion states. Gaussian kernel regression and linear extrapolation were used to predict PCA coefficients of future motion states for breathing motion (340 ms ahead of acquisition) and slow drifting motion (8.5 s ahead of acquisition) respectively. k-space data from the remaining five-minutes was used to compare ground truth motions states obtained from retrospective reconstruction/deformation with predictions. Median distances between predicted and ground truth centroid positions of gross tumor volume (GTV) and organs at risk (OARs) were less than 1 mm on average. 95- percentile Hausdorff distances between predicted and ground truth GTV contours of various breathing motions states were 2 mm on average, which was smaller than the imaging resolution and 95-percentile Hausdorff distances between predicted and ground truth OAR contours of different slow drifting motion states were less than 0.2 mm. These results suggest that multi-temporal resolution motion models are capable of volumetric predictions of breathing and slow drifting motion with sufficient accuracy and temporal resolution for MRI-based tracking, and thus have potential for supporting MRI-guided abdominal radiotherapy.

Technical Challenges of Real-Time Adaptive MR-Guided Radiotherapy

In the past few years, radiotherapy (RT) has experienced a major technological innovation with the development of hybrid machines combining magnetic resonance (MR) imaging and linear accelerators. This new technology for MR-guided cancer treatment has the potential to revolutionize the field of adaptive RT due to the opportunity to provide high-resolution, real-time MR imaging before and during treatment application. However, from a technical point of view, several challenges remain which need to be tackled to ensure safe and robust real-time adaptive MR-guided RT delivery. In this manuscript, several technical challenges to MR-guided RT are discussed. Starting with magnetic field strength tradeoffs, the potential and limitations for purely MR-based RT workflows are discussed. Furthermore, the current status of real-time 3D MR imaging and its potential for real-time RT are summarized. Finally, the potential of quantitative MR imaging for future biological RT adaptation is highlighted.

A motion prediction confidence estimation framework for prediction-based radiotherapy gating

PURPOSE

Motion prediction can compensate for latency in image-guided radiotherapy and has been an active area of research. However, motion predictions are subject to error and variations. We have developed and evaluated a novel motion prediction confidence estimation framework to improve the efficacy and robustness of prediction-based radiotherapy gating decision-making. The specific scenario of adaptive gating in magnetic resonance imaging (MRI)-guided radiotherapy is studied as an example, but the method generalizes to other modalities and motion management setups.

METHODS

The proposed prediction confidence estimator is based on a generic training/testing paradigm and consists of a weighted combination of three components: the prediction model's goodness of fit, variation in the prediction using a leave-one-out process and the velocity of the tracked target. Roughly, these terms quantify respectively the consistency between prediction and the training data, the robustness of model inference, and the stability due to target speed. The weight parameters and the action level in triggering beam-off decision are optimized. The method is assessed and validated in 8 healthy volunteer and 13 patient studies using a 0.35T MRI-guided radiotherapy system predicting 0.25-0.33 s ahead. The effect of the action level on the predicted gating decision accuracy, beam-on positive predictive value (PPV) and median distance between the predicted and ground-truth target centroids were evaluated. Statistical significance was evaluated using a paired t-test. The tradeoff between these performance metrics and gating duty cycle was assessed.

RESULTS

Use of the confidence estimator threshold increased gating accuracy by up to 2.42%, increased PPV by up to 3.00%, and reduced the median centroid distance up to 0.28 mm. The confidence estimator threshold on average increased gating accuracy to 96.5% (P = 2.08 × 10^-4 ), increased PPV to 96.7% (P = 1.46 × 10^-5 ), reduced the median centroid distance to 0.54 mm (P = 1.71 × 10^-5 ) at the cost of reducing the gating duty cycle by 14.3% to 48.5%. Hyperparameter tuning revealed that contrary to intuition, the velocity term offered only minimal performance improvement in some cases but also introduced potential stability issues. The combination of goodness of fit and leave-one-out prediction variation provided the most effective confidence estimator, yielding universally better performance in gating decisions.

CONCLUSION

Confidence estimation utilizing prediction model fitness criterion and validation principles can complement prediction methods to guide MRI-guided radiotherapy gating. Results from both volunteer and patient studies showed improved gating quality.