A generalized framework unifying image registration and respiratory motion models and incorporating image reconstruction, for partial image data or full images

Data driven surrogate signal extraction for dynamic PET using selective PCA: time windows versus the combination of components

Objective.Respiratory motion correction is beneficial in positron emission tomography (PET), as it can reduce artefacts caused by motion and improve quantitative accuracy. Methods of motion correction are commonly based on a respiratory trace obtained through an external device (like the real time position management system) or a data driven method, such as those based on dimensionality reduction techniques (for instance principal component analysis (PCA)). PCA itself being a linear transformation to the axis of greatest variation. Data driven methods have the advantage of being non-invasive, and can be performed post-acquisition. However, their main downside being that they are adversely affected by the tracer kinetics of the dynamic PET acquisition. Therefore, they are mostly limited to static PET acquisitions. This work seeks to extend on existing PCA-based data-driven motion correction methods, to allow for their applicability to dynamic PET imaging.Approach.The methods explored in this work include; a moving window approach (similar to the Kinetic Respiratory Gating method from Schleyeret al(2014)), extrapolation of the principal component from later time points to earlier time points, and a method to score, select, and combine multiple respiratory components. The resulting respiratory traces were evaluated on 22 data sets from a dynamic [18F]-FDG study on patients with idiopathic pulmonary fibrosis. This was achieved by calculating their correlation with a surrogate signal acquired using a real time position management system.Main results.The results indicate that all methods produce better surrogate signals than when applying conventional PCA to dynamic data (for instance, a higher correlation with a gold standard respiratory trace). Extrapolating a late time point principal component produced more promising results than using a moving window. Scoring, selecting, and combining components held benefits over all other methods.Significance.This work allows for the extraction of a surrogate signal from dynamic PET data earlier in the acquisition and with a greater accuracy than previous work. This potentially allows for numerous other methods (for instance, respiratory motion correction) to be applied to this data (when they otherwise could not be previously used).

4D-Precise: Learning-based 3D motion estimation and high temporal resolution 4DCT reconstruction from treatment 2D+t X-ray projections

BACKGROUND AND OBJECTIVE

In radiotherapy treatment planning, respiration-induced motion introduces uncertainty that, if not appropriately considered, could result in dose delivery problems. 4D cone-beam computed tomography (4D-CBCT) has been developed to provide imaging guidance by reconstructing a pseudo-motion sequence of CBCT volumes through binning projection data into breathing phases. However, it suffers from artefacts and erroneously characterizes the averaged breathing motion. Furthermore, conventional 4D-CBCT can only be generated post-hoc using the full sequence of kV projections after the treatment is complete, limiting its utility. Hence, our purpose is to develop a deep-learning motion model for estimating 3D+t CT images from treatment kV projection series.

METHODS

We propose an end-to-end learning-based 3D motion modelling and 4DCT reconstruction model named 4D-Precise, abbreviated from Probabilistic reconstruction of image sequences from CBCT kV projections. The model estimates voxel-wise motion fields and simultaneously reconstructs a 3DCT volume at any arbitrary time point of the input projections by transforming a reference CT volume. Developing a Torch-DRR module, it enables end-to-end training by computing Digitally Reconstructed Radiographs (DRRs) in PyTorch. During training, DRRs with matching projection angles to the input kVs are automatically extracted from reconstructed volumes and their structural dissimilarity to inputs is penalised. We introduced a novel loss function to regulate spatio-temporal motion field variations across the CT scan, leveraging planning 4DCT for prior motion distribution estimation.

RESULTS

The model is trained patient-specifically using three kV scan series, each including over 1200 angular/temporal projections, and tested on three other scan series. Imaging data from five patients are analysed here. Also, the model is validated on a simulated paired 4DCT-DRR dataset created using the Surrogate Parametrised Respiratory Motion Modelling (SuPReMo). The results demonstrate that the reconstructed volumes by 4D-Precise closely resemble the ground-truth volumes in terms of Dice, volume similarity, mean contour distance, and Hausdorff distance, whereas 4D-Precise achieves smoother deformations and fewer negative Jacobian determinants compared to SuPReMo.

CONCLUSIONS

Unlike conventional 4DCT reconstruction techniques that ignore breath inter-cycle motion variations, the proposed model computes both intra-cycle and inter-cycle motions. It represents motion over an extended timeframe, covering several minutes of kV scan series.

Respiratory motion modelling for MR-guided lung cancer radiotherapy: model development and geometric accuracy evaluation

Objective.Respiratory motion of lung tumours and adjacent structures is challenging for radiotherapy. Online MR-imaging cannot currently provide real-time volumetric information of the moving patient anatomy, therefore limiting precise dose delivery, delivered dose reconstruction, and downstream adaptation methods.Approach.We tailor a respiratory motion modelling framework towards an MR-Linac workflow to estimate the time-resolved 4D motion from real-time data. We develop a multi-slice acquisition scheme which acquires thick, overlapping 2D motion-slices in different locations and orientations, interleaved with 2D surrogate-slices from a fixed location. The framework fits a motion model directly to the input data without the need for sorting or binning to account for inter- and intra-cycle variation of the breathing motion. The framework alternates between model fitting and motion-compensated super-resolution image reconstruction to recover a high-quality motion-free image and a motion model. The fitted model can then estimate the 4D motion from 2D surrogate-slices. The framework is applied to four simulated anthropomorphic datasets and evaluated against known ground truth anatomy and motion. Clinical applicability is demonstrated by applying our framework to eight datasets acquired on an MR-Linac from four lung cancer patients.Main results.The framework accurately reconstructs high-quality motion-compensated 3D images with 2 mm3isotropic voxels. For the simulated case with the largest target motion, the motion model achieved a mean deformation field error of 1.13 mm. For the patient cases residual error registrations estimate the model error to be 1.07 mm (1.64 mm), 0.91 mm (1.32 mm), and 0.88 mm (1.33 mm) in superior-inferior, anterior-posterior, and left-right directions respectively for the building (application) data.Significance.The motion modelling framework estimates the patient motion with high accuracy and accurately reconstructs the anatomy. The image acquisition scheme can be flexibly integrated into an MR-Linac workflow whilst maintaining the capability of online motion-management strategies based on cine imaging such as target tracking and/or gating.

Surrogate-driven respiratory motion model for projection-resolved motion estimation and motion compensated cone-beam CT reconstruction from unsorted projection data

Objective.As the most common solution to motion artefact for cone-beam CT (CBCT) in radiotherapy, 4DCBCT suffers from long acquisition time and phase sorting error. This issue could be addressed if the motion at each projection could be known, which is a severely ill-posed problem. This study aims to obtain the motion at each time point and motion-free image simultaneously from unsorted projection data of a standard 3DCBCT scan.Approach.Respiration surrogate signals were extracted by the Intensity Analysis method. A general framework was then deployed to fit a surrogate-driven motion model that characterized the relation between the motion and surrogate signals at each time point. Motion model fitting and motion compensated reconstruction were alternatively and iteratively performed. Stochastic subset gradient based method was used to significantly reduce the computation time. The performance of our method was comprehensively evaluated through digital phantom simulation and also validated on clinical scans from six patients.Results.For digital phantom experiments, motion models fitted with ground-truth or extracted surrogate signals both achieved a much lower motion estimation error and higher image quality, compared with non motion-compensated results.For the public SPARE Challenge datasets, more clear lung tissues and less blurry diaphragm could be seen in the motion compensated reconstruction, comparable to the benchmark 4DCBCT images but with a higher temporal resolution. Similar results were observed for two real clinical 3DCBCT scans.Significance.The motion compensated reconstructions and motion models produced by our method will have direct clinical benefit by providing more accurate estimates of the delivered dose and ultimately facilitating more accurate radiotherapy treatments for lung cancer patients.

Ultra-fast multi-parametric 4D-MRI image reconstruction for real-time applications using a downsampling-invariant deformable registration (D2R) model

BACKGROUND AND PURPOSE

Motion estimation from severely downsampled 4D-MRI is essential for real-time imaging and tumor tracking. This simulation study developed a novel deep learning model for simultaneous MR image reconstruction and motion estimation, named the Downsampling-Invariant Deformable Registration (D2R) model.

MATERIALS AND METHODS

Forty-three patients undergoing radiotherapy for liver tumors were recruited for model training and internal validation. Five prospective patients from another center were recruited for external validation. Patients received 4D-MRI scans and 3D MRI scans. The 4D-MRI was retrospectively down-sampled to simulate real-time acquisition. Motion estimation was performed using the proposed D2R model. The accuracy and robustness of the proposed D2R model and baseline methods, including Demons, Elastix, the parametric total variation (pTV) algorithm, and VoxelMorph, were compared. High-quality (HQ) 4D-MR images were also constructed using the D2R model for real-time imaging feasibility verification. The image quality and motion accuracy of the constructed HQ 4D-MRI were evaluated.

RESULTS

The D2R model showed significantly superior and robust registration performance than all the baseline methods at downsampling factors up to 500. HQ T1-weighted and T2-weighted 4D-MR images were also successfully constructed with significantly improved image quality, sub-voxel level motion error, and real-time efficiency. External validation demonstrated the robustness and generalizability of the technique.

CONCLUSION

In this study, we developed a novel D2R model for deformation estimation of downsampled 4D-MR images. HQ 4D-MR images were successfully constructed using the D2R model. This model may expand the clinical implementation of 4D-MRI for real-time motion management during liver cancer treatment.

Gaussian Processes for real-time 3D motion and uncertainty estimation during MR-guided radiotherapy

Respiratory motion during radiotherapy causes uncertainty in the tumor's location, which is typically addressed by an increased radiation area and a decreased dose. As a result, the treatments' efficacy is reduced. The recently proposed hybrid MR-linac scanner holds the promise to efficiently deal with such respiratory motion through real-time adaptive MR-guided radiotherapy (MRgRT). For MRgRT, motion-fields should be estimated from MR-data and the radiotherapy plan should be adapted in real-time according to the estimated motion-fields. All of this should be performed with a total latency of maximally 200 ms, including data acquisition and reconstruction. A measure of confidence in such estimated motion-fields is highly desirable, for instance to ensure the patient's safety in case of unexpected and undesirable motion. In this work, we propose a framework based on Gaussian Processes to infer 3D motion-fields and uncertainty maps in real-time from only three readouts of MR-data. We demonstrated an inference frame rate up to 69 Hz including data acquisition and reconstruction, thereby exploiting the limited amount of required MR-data. Additionally, we designed a rejection criterion based on the motion-field uncertainty maps to demonstrate the framework's potential for quality assurance. The framework was validated in silico and in vivo on healthy volunteer data (n=5) acquired using an MR-linac, thereby taking into account different breathing patterns and controlled bulk motion. Results indicate end-point-errors with a 75th percentile below 1 mm in silico, and a correct detection of erroneous motion estimates with the rejection criterion. Altogether, the results show the potential of the framework for application in real-time MR-guided radiotherapy with an MR-linac.

Breathing-Adapted Imaging Techniques for Rapid 4-Dimensional Lung Tomosynthesis

Purpose

This article presents enhancements to a 4-dimensional (4D) lung digital tomosynthesis (DTS) model introduced in a 2018 paper. That model was proposed as an adjunct to 4D computed tomography (4DCT) to improve tumor localization through artifact reduction achieved by imaging the entire lung in all projections, reducing the projection collection time duration for each phase compared with 4DCT, and requiring only a single-breath cycle to capture all phases. This is applicable to SABR treatment planning. Enhancements comprise customized patient 4D-DTS x-ray scanning parameters.

Methods and Materials

Imaging parameters derived with the 4D-DTS model were arc duration, frames per second, pulse duration, and tube current normalized to single-chest radiographic milliampere-seconds (mA/mAs_AEC). Optimized phase-specific DTS projections imaging parameters were derived for volunteer respiration-tracking surrogate waveforms and for sinusoidal waveforms. These parameters are temporally matched to the respiratory surrogate waveform and presented as continuous data plots during a period of 20 seconds. Comparison is made between surrogate excursions during a single-phase CT and 4D-DTS reconstructions.

Results

4D-DTS imaging techniques were customized to volunteer respiratory waveforms and sinusoidal waveforms. Technique settings at the highest velocity portions of the volunteer waveforms were arc duration 0.066 seconds, frame rate 921 Hz, pulse duration 1.076 ms, and normalized tube current 76.2 s^-1. Technique settings at the highest velocity portions of the sinusoidal waveforms were arc duration 0.029 seconds, frame rate 2074 Hz, pulse duration 0.472 ms, and normalized tube current 173.6 s^-1. Sinusoidal surrogate excursion distance at the highest velocity portion of the waveform during a CT rotation of 0.5 seconds ranged from 2.68 to 21.09 mm, all greater than the limiting excursion distance chosen in the 4D-DTS model.

Conclusions

4D-DTS image technique settings can be customized to individual patient breathing patterns so that captured range of motion satisfies an operator-selected value.

A personalized image-guided intervention system for peripheral lung cancer on patient-specific respiratory motion model

PURPOSE

Due to respiratory motion, precise tracking of lung nodule movement is a persistent challenge for guiding percutaneous lung biopsy during image-guided intervention. We developed an automated image-guided system incorporating effective and robust tracking algorithms to address this challenge. Accurate lung motion prediction and personalized image-guided intervention are the key technological contributions of this work.

METHODS

A patient-specific respiratory motion model is developed to predict pulmonary movements of individual patients. It is based on the relation between the artificial 4D CT and corresponding positions tracked by position sensors attached on the chest using an electromagnetic (EM) tracking system. The 4D CT image of the thorax during breathing is calculated through deformable registration of two 3D CT scans acquired at inspiratory and expiratory breath-hold. The robustness and accuracy of the image-guided intervention system were assessed on a static thorax phantom under different clinical parametric combinations.

RESULTS

Real 4D CT images of ten patients were used to evaluate the accuracy of the respiratory motion model. The mean error of the model in different breathing phases was 1.59 ± 0.66 mm. Using a static thorax phantom, we achieved an average targeting accuracy of 3.18 ± 1.2 mm across 50 independent tests with different intervention parameters. The positive results demonstrate the robustness and accuracy of our system for personalized lung cancer intervention.

CONCLUSIONS

The proposed system integrates a patient-specific respiratory motion compensation model to reduce the effect of respiratory motion during percutaneous lung biopsy and help interventional radiologists target the lesion efficiently. Our preclinical studies indicate that the image-guided system has the ability to accurately predict and track lung nodules of individual patients and has the potential for use in the diagnosis and treatment of early stage lung cancer.

Real-Time Non-Rigid 3D Respiratory Motion Estimation for MR-Guided Radiotherapy Using MR-MOTUS

The MR-Linac is a combination of an MR-scanner and radiotherapy linear accelerator (Linac) which holds the promise to increase the precision of radiotherapy treatments with MR-guided radiotherapy by monitoring motion during radiotherapy with MRI, and adjusting the radiotherapy plan accordingly. Optimal MR-guidance for respiratory motion during radiotherapy requires MR-based 3D motion estimation with a latency of 200-500 ms. Currently this is still challenging since typical methods rely on MR-images, and are therefore limited by the 3D MR-imaging latency. In this work, we present a method to perform non-rigid 3D respiratory motion estimation with 170 ms latency, including both acquisition and reconstruction. The proposed method called real-time low-rank MR-MOTUS reconstructs motion-fields directly from k -space data, and leverages an explicit low-rank decomposition of motion-fields to split the large scale 3D+t motion-field reconstruction problem posed in our previous work into two parts: (I) a medium-scale offline preparation phase and (II) a small-scale online inference phase which exploits the results of the offline phase for real-time computations. The method was validated on free-breathing data of five volunteers, acquired with a 1.5T Elekta Unity MR-Linac. Results show that the reconstructed 3D motion-field are anatomically plausible, highly correlated with a self-navigation motion surrogate ( R=0.975 ±0.0110 ), and can be reconstructed with a total latency of 170 ms that is sufficient for real-time MR-guided abdominal radiotherapy.

PET respiratory motion correction: quo vadis?

Positron emission tomography (PET) respiratory motion correction has been a subject of great interest for the last twenty years, prompted mainly by the development of multimodality imaging devices such as PET/computed tomography (CT) and PET/magnetic resonance imaging (MRI). PET respiratory motion correction involves a number of steps including acquisition synchronization, motion estimation and finally motion correction. The synchronization steps include the use of different external device systems or data driven approaches which have been gaining ground over the last few years. Patient specific or generic motion models using the respiratory synchronized datasets can be subsequently derived and used for correction either in the image space or within the image reconstruction process. Similar overall approaches can be considered and have been proposed for both PET/CT and PET/MRI devices. Certain variations in the case of PET/MRI include the use of MRI specific sequences for the registration of respiratory motion information. The proposed review includes a comprehensive coverage of all these areas of development in field of PET respiratory motion for different multimodality imaging devices and approaches in terms of synchronization, estimation and subsequent motion correction. Finally, a section on perspectives including the potential clinical usage of these approaches is included.

Improving motion-mask segmentation in thoracic CT with multiplanar U-nets

PURPOSE

Motion-mask segmentation from thoracic computed tomography (CT) images is the process of extracting the region that encompasses lungs and viscera, where large displacements occur during breathing. It has been shown to help image registration between different respiratory phases. This registration step is, for example, useful for radiotherapy planning or calculating local lung ventilation. Knowing the location of motion discontinuity, that is, sliding motion near the pleura, allows a better control of the registration preventing unrealistic estimates. Nevertheless, existing methods for motion-mask segmentation are not robust enough to be used in clinical routine. This article shows that it is feasible to overcome this lack of robustness by using a lightweight deep-learning approach usable on a standard computer, and this even without data augmentation or advanced model design.

METHODS

A convolutional neural-network architecture with three 2D U-nets for the three main orientations (sagittal, coronal, axial) was proposed. Predictions generated by the three U-nets were combined by majority voting to provide a single 3D segmentation of the motion mask. The networks were trained on a database of nonsmall cell lung cancer 4D CT images of 43 patients. Training and evaluation were done with a K-fold cross-validation strategy. Evaluation was based on a visual grading by two experts according to the appropriateness of the segmented motion mask for the registration task, and on a comparison with motion masks obtained by a baseline method using level sets. A second database (76 CT images of patients with early-stage COVID-19), unseen during training, was used to assess the generalizability of the trained neural network.

RESULTS

The proposed approach outperformed the baseline method in terms of quality and robustness: the success rate increased from 53 % to 79 % without producing any failure. It also achieved a speed-up factor of 60 with GPU, or 17 with CPU. The memory footprint was low: less than 5 GB GPU RAM for training and less than 1 GB GPU RAM for inference. When evaluated on a dataset with images differing by several characteristics (CT device, pathology, and field of view), the proposed method improved the success rate from 53 % to 83 % .

CONCLUSION

With 5-s processing time on a mid-range GPU and success rates around 80 % , the proposed approach seems fast and robust enough to be routinely used in clinical practice. The success rate can be further improved by incorporating more diversity in training data via data augmentation and additional annotated images from different scanners and diseases. The code and trained model are publicly available.

Projection-based dynamic tomography

Objective. This paper proposes a 4D dynamic tomography framework that allows a moving sample to be imaged in a tomograph as well as the associated space-time kinematics to be measured with nothing more than a single conventional x-ray scan.Approach. The method exploits the consistency of the projection/reconstruction operations through a multi-scale procedure. The procedure is composed of two main parts solved alternatively: a motion-compensated reconstruction algorithm and a projection-based measurement procedure that estimates the displacement field directly on each projection.Main results. The method is applied to two studies: a numerical simulation of breathing from chest computed tomography (4D-CT) and a clinical cone-beam CT of a breathing patient acquired for image guidance of radiotherapy. The reconstructed volume, initially blurred by the motion, is cleaned from motion artifacts.Significance. Applying the proposed approach results in an improved reconstruction quality showing sharper edges and finer details.

Population-based 3D respiratory motion modelling from convolutional autoencoders for 2D ultrasound-guided radiotherapy

Radiotherapy is a widely used treatment modality for various types of cancers. A challenge for precise delivery of radiation to the treatment site is the management of internal motion caused by the patient's breathing, especially around abdominal organs such as the liver. Current image-guided radiation therapy (IGRT) solutions rely on ionising imaging modalities such as X-ray or CBCT, which do not allow real-time target tracking. Ultrasound imaging (US) on the other hand is relatively inexpensive, portable and non-ionising. Although 2D US can be acquired at a sufficient temporal frequency, it doesn't allow for target tracking in multiple planes, while 3D US acquisitions are not adapted for real-time. In this work, a novel deep learning-based motion modelling framework is presented for ultrasound IGRT. Our solution includes an image similarity-based rigid alignment module combined with a deep deformable motion model. Leveraging the representational capabilities of convolutional autoencoders, our deformable motion model associates complex 3D deformations with 2D surrogate US images through a common learned low dimensional representation. The model is trained on a variety of deformations and anatomies which enables it to generate the 3D motion experienced by the liver of a previously unseen subject. During inference, our framework only requires two pre-treatment 3D volumes of the liver at extreme breathing phases and a live 2D surrogate image representing the current state of the organ. In this study, the presented model is evaluated on a 3D+t US data set of 20 volunteers based on image similarity as well as anatomical target tracking performance. We report results that surpass comparable methodologies in both metric categories with a mean tracking error of 3.5±2.4 mm, demonstrating the potential of this technique for IGRT.

Motion estimation and correction for simultaneous PET/MR using SIRF and CIL

SIRF is a powerful PET/MR image reconstruction research tool for processing data and developing new algorithms. In this research, new developments to SIRF are presented, with focus on motion estimation and correction. SIRF's recent inclusion of the adjoint of the resampling operator allows gradient propagation through resampling, enabling the MCIR technique. Another enhancement enabled registering and resampling of complex images, suitable for MRI. Furthermore, SIRF's integration with the optimization library CIL enables the use of novel algorithms. Finally, SPM is now supported, in addition to NiftyReg, for registration. Results of MR and PET MCIR reconstructions are presented, using FISTA and PDHG, respectively. These demonstrate the advantages of incorporating motion correction and variational and structural priors. This article is part of the theme issue 'Synergistic tomographic image reconstruction: part 2'.

Artificial Intelligence in magnetic Resonance guided Radiotherapy: Medical and physical considerations on state of art and future perspectives

Over the last years, technological innovation in Radiotherapy (RT) led to the introduction of Magnetic Resonance-guided RT (MRgRT) systems. Due to the higher soft tissue contrast compared to on-board CT-based systems, MRgRT is expected to significantly improve the treatment in many situations. MRgRT systems may extend the management of inter- and intra-fraction anatomical changes, offering the possibility of online adaptation of the dose distribution according to daily patient anatomy and to directly monitor tumor motion during treatment delivery by means of a continuous cine MR acquisition. Online adaptive treatments require a multidisciplinary and well-trained team, able to perform a series of operations in a safe, precise and fast manner while the patient is waiting on the treatment couch. Artificial Intelligence (AI) is expected to rapidly contribute to MRgRT, primarily by safely and efficiently automatising the various manual operations characterizing online adaptive treatments. Furthermore, AI is finding relevant applications in MRgRT in the fields of image segmentation, synthetic CT reconstruction, automatic (on-line) planning and the development of predictive models based on daily MRI. This review provides a comprehensive overview of the current AI integration in MRgRT from a medical physicist's perspective. Medical physicists are expected to be major actors in solving new tasks and in taking new responsibilities: their traditional role of guardians of the new technology implementation will change with increasing emphasis on the managing of AI tools, processes and advanced systems for imaging and data analysis, gradually replacing many repetitive manual tasks.

Porcine lung phantom-based validation of estimated 4D-MRI using orthogonal cine imaging for low-field MR-Linacs

Real-time motion monitoring of lung tumors with low-field magnetic resonance imaging-guided linear accelerators (MR-Linacs) is currently limited to sagittal 2D cine magnetic resonance imaging (MRI). To provide input data for improved intrafractional and interfractional adaptive radiotherapy, the 4D anatomy has to be inferred from data with lower dimensionality. The purpose of this study was to experimentally validate a previously proposed propagation method that provides continuous time-resolved estimated 4D-MRI based on orthogonal cine MRI for a low-field MR-Linac. Ex vivo porcine lungs were injected with artificial nodules and mounted in a dedicated phantom that allows for the simulation of periodic and reproducible breathing motion. The phantom was scanned with a research version of a commercial 0.35 T MR-Linac. Respiratory-correlated 4D-MRI were reconstructed and served as ground truth images. Series of interleaved orthogonal slices in sagittal and coronal orientation, intersecting the injected targets, were acquired at 7.3 Hz. Estimated 4D-MRI at 3.65 Hz were created in post-processing using the propagation method and compared to the ground truth 4D-MRI. Eight datasets at different breathing frequencies and motion amplitudes were acquired for three porcine lungs. The overall median (95[Formula: see text] percentile) deviation between ground truth and estimated deformation vector fields was 2.3 mm (5.7 mm), corresponding to 0.7 (1.6) times the in-plane imaging resolution (3.5 × 3.5 mm²). Median (95[Formula: see text] percentile) estimated nodule position errors were 1.5 mm (3.8 mm) for nodules intersected by orthogonal slices and 2.1 mm (7.1 mm) for nodules located more than 2 cm away from either of the orthogonal slices. The estimation error depended on the breathing phase, the motion amplitude and the location of the estimated position with respect to the orthogonal slices. By using the propagation method, the 4D motion within the porcine lung phantom could be accurately and robustly estimated. The method could provide valuable information for treatment planning, real-time motion monitoring, treatment adaptation, and post-treatment evaluation of MR-guided radiotherapy treatments.

Clinical practice vs. state-of-the-art research and future visions: Report on the 4D treatment planning workshop for particle therapy - Edition 2018 and 2019

The 4D Treatment Planning Workshop for Particle Therapy, a workshop dedicated to the treatment of moving targets with scanned particle beams, started in 2009 and since then has been organized annually. The mission of the workshop is to create an informal ground for clinical medical physicists, medical physics researchers and medical doctors interested in the development of the 4D technology, protocols and their translation into clinical practice. The 10th and 11th editions of the workshop took place in Sapporo, Japan in 2018 and Krakow, Poland in 2019, respectively. This review report from the Sapporo and Krakow workshops is structured in two parts, according to the workshop programs. The first part comprises clinicians and physicists review of the status of 4D clinical implementations. Corresponding talks were given by speakers from five centers around the world: Maastro Clinic (The Netherlands), University Medical Center Groningen (The Netherlands), MD Anderson Cancer Center (United States), University of Pennsylvania (United States) and The Proton Beam Therapy Center of Hokkaido University Hospital (Japan). The second part is dedicated to novelties in 4D research, i.e. motion modelling, artificial intelligence and new technologies which are currently being investigated in the radiotherapy field.

Nonrigid 3D motion estimation at high temporal resolution from prospectively undersampled k-space data using low-rank MR-MOTUS

Purpose

With the recent introduction of the MR‐LINAC, an MR‐scanner combined with a radiotherapy LINAC, MR‐based motion estimation has become of increasing interest to (retrospectively) characterize tumor and organs‐at‐risk motion during radiotherapy. To this extent, we introduce low‐rank MR‐MOTUS, a framework to retrospectively reconstruct time‐resolved nonrigid 3D+t motion fields from a single low‐resolution reference image and prospectively undersampled k‐space data acquired during motion.

Theory

Low‐rank MR‐MOTUS exploits spatiotemporal correlations in internal body motion with a low‐rank motion model, and inverts a signal model that relates motion fields directly to a reference image and k‐space data. The low‐rank model reduces the degrees‐of‐freedom, memory consumption, and reconstruction times by assuming a factorization of space‐time motion fields in spatial and temporal components.

Methods

Low‐rank MR‐MOTUS was employed to estimate motion in 2D/3D abdominothoracic scans and 3D head scans. Data were acquired using golden‐ratio radial readouts. Reconstructed 2D and 3D respiratory motion fields were, respectively, validated against time‐resolved and respiratory‐resolved image reconstructions, and the head motion against static image reconstructions from fully sampled data acquired right before and right after the motion.

Results

Results show that 2D+t respiratory motion can be estimated retrospectively at 40.8 motion fields per second, 3D+t respiratory motion at 7.6 motion fields per second and 3D+t head‐neck motion at 9.3 motion fields per second. The validations show good consistency with image reconstructions.

Conclusions

The proposed framework can estimate time‐resolved nonrigid 3D motion fields, which allows to characterize drifts and intra and inter‐cycle patterns in breathing motion during radiotherapy, and could form the basis for real‐time MR‐guided radiotherapy.

Deep learning in medical image registration: a review

This paper presents a review of deep learning (DL)-based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potential. We provided a comprehensive comparison among DL-based methods for lung and brain registration using benchmark datasets. Lastly, we analyzed the statistics of all the cited works from various aspects, revealing the popularity and future trend of DL-based medical image registration.

A modern review of the uncertainties in volumetric imaging of respiratory-induced target motion in lung radiotherapy

Radiotherapy has become a critical component for the treatment of all stages and types of lung cancer, often times being the primary gateway to a cure. However, given that radiation can cause harmful side effects depending on how much surrounding healthy tissue is exposed, treatment of the lung can be particularly challenging due to the presence of moving targets. Careful implementation of every step in the radiotherapy process is absolutely integral for attaining optimal clinical outcomes. With the advent and now widespread use of stereotactic body radiation therapy (SBRT), where extremely large doses are delivered, accurate, and precise dose targeting is especially vital to achieve an optimal risk to benefit ratio. This has largely become possible due to the rapid development of image-guided technology. Although imaging is critical to the success of radiotherapy, it can often be plagued with uncertainties due to respiratory-induced target motion. There has and continues to be an immense research effort aimed at acknowledging and addressing these uncertainties to further our abilities to more precisely target radiation treatment. Thus, the goal of this article is to provide a detailed review of the prevailing uncertainties that remain to be investigated across the different imaging modalities, as well as to highlight the more modern solutions to imaging motion and their role in addressing the current challenges.

Evaluation of MRI-derived surrogate signals to model respiratory motion

An MR-Linac can provide motion information of tumour and organs-at-risk before, during, and after beam delivery. However, MR imaging cannot provide real-time high-quality volumetric images which capture breath-to-breath variability of respiratory motion. Surrogate-driven motion models relate the motion of the internal anatomy to surrogate signals, thus can estimate the 3D internal motion from these signals. Internal surrogate signals based on patient anatomy can be extracted from 2D cine-MR images, which can be acquired on an MR-Linac during treatment, to build and drive motion models. In this paper we investigate different MRI-derived surrogate signals, including signals generated by applying principal component analysis to the image intensities, or control point displacements derived from deformable registration of the 2D cine-MR images. We assessed the suitability of the signals to build models that can estimate the motion of the internal anatomy, including sliding motion and breath-to-breath variability. We quantitatively evaluated the models by estimating the 2D motion in sagittal and coronal slices of 8 lung cancer patients, and comparing them to motion measurements obtained from image registration. For sagittal slices, using the first and second principal components on the control point displacements as surrogate signals resulted in the highest model accuracy, with a mean error over patients around 0.80 mm which was lower than the in-plane resolution. For coronal slices, all investigated signals except the skin signal produced mean errors over patients around 1 mm. These results demonstrate that surrogate signals derived from 2D cine-MR images, including those generated by applying principal component analysis to the image intensities or control point displacements, can accurately model the motion of the internal anatomy within a single sagittal or coronal slice. This implies the signals should also be suitable for modelling the 3D respiratory motion of the internal anatomy.

The noise navigator for MRI-guided radiotherapy: an independent method to detect physiological motion

Motion is problematic during radiotherapy as it could lead to potential underdosage of the tumor, and/or overdosage in organs-at-risk. A solution is adaptive radiotherapy guided by magnetic resonance imaging (MRI). MRI allows for imaging of target volumes and organs-at-risk before and during treatment delivery with superb soft tissue contrast in any desired orientation, enabling motion management by means of (real-time) adaptive radiotherapy. The noise navigator, which is independent of the MR signal, could serve as a secondary motion detection method in synergy with MR imaging. The feasibility of respiratory motion detection by means of the noise navigator was demonstrated previously. Furthermore, from electromagnetic simulations we know that the noise navigator is sensitive to tissue displacement and thus could in principle be used for the detection of various types of motion. In this study we demonstrate the detection of various types of motion for three anatomical use cases of MRI-guided radiotherapy, i.e. torso (bulk movement and variable breathing), head-and-neck (swallowing) and cardiac. Furthermore, it is shown that the noise navigator can detect bulk movement, variable breathing and swallowing on a hybrid 1.5 T MRI-linac system. Cardiac activity detection through the noise navigator seems feasible in an MRI-guided radiotherapy setting, but needs further optimization. The noise navigator is a versatile and fast (millisecond temporal resolution) motion detection method independent of MR signal that could serve as an independent verification method to detect the occurrence of motion in synergy with real-time MRI-guided radiotherapy.

MR-MOTUS: model-based non-rigid motion estimation for MR-guided radiotherapy using a reference image and minimal k-space data

Time-resolved motion estimation from MRI data has received an increasing amount of interest due to the advent of the MR-Linac. The combination of an MRI scanner and a linear accelerator enables radiation plan adaptation based on internal organ motion estimated from MRI data. However, time-resolved estimation of this motion from MRI data still remains a challenge. In light of this application, we propose MR-MOTUS, a framework to estimate non-rigid 3D motion from minimal k-space data. MR-MOTUS consists of two main components: (1) a signal model that explicitly relates the k-space signal of a deforming object to non-rigid motion-fields and a reference image, and (2) model-based reconstructions of the non-rigid motion-fields directly from k-space data. Using an a priori available reference image and the fact that internal body motion exhibits a high level of spatial correlation, we represent the motion-fields in a low-dimensional space and reconstruct them from minimal k-space data that can be acquired very rapidly. The signal model is validated through numerical experiments with a digital 3D phantom and motion-fields are reconstructed from retrospectively undersampled in vivo head and abdomen data using various undersampling strategies. A comparison is made with state-of-the-art image registration performed on images reconstructed from the same undersampled data. Results show that MR-MOTUS reconstructs in vivo 3D rigid head motion from 474-fold retrospectively downsampled k-space data, and in vivo non-rigid 3D respiratory motion from 63-fold retrospectively undersampled k-space data. Preliminary results on prospectively undersampled data acquired with a 2D golden angle acquisition during free-breathing demonstrate the practical feasibility of the method.

Real-time intrafraction motion monitoring in external beam radiotherapy

Radiotherapy (RT) aims to deliver a spatially conformal dose of radiation to tumours while maximizing the dose sparing to healthy tissues. However, the internal patient anatomy is constantly moving due to respiratory, cardiac, gastrointestinal and urinary activity. The long term goal of the RT community to 'see what we treat, as we treat' and to act on this information instantaneously has resulted in rapid technological innovation. Specialized treatment machines, such as robotic or gimbal-steered linear accelerators (linac) with in-room imaging suites, have been developed specifically for real-time treatment adaptation. Additional equipment, such as stereoscopic kilovoltage (kV) imaging, ultrasound transducers and electromagnetic transponders, has been developed for intrafraction motion monitoring on conventional linacs. Magnetic resonance imaging (MRI) has been integrated with cobalt treatment units and more recently with linacs. In addition to hardware innovation, software development has played a substantial role in the development of motion monitoring methods based on respiratory motion surrogates and planar kV or Megavoltage (MV) imaging that is available on standard equipped linacs. In this paper, we review and compare the different intrafraction motion monitoring methods proposed in the literature and demonstrated in real-time on clinical data as well as their possible future developments. We then discuss general considerations on validation and quality assurance for clinical implementation. Besides photon RT, particle therapy is increasingly used to treat moving targets. However, transferring motion monitoring technologies from linacs to particle beam lines presents substantial challenges. Lessons learned from the implementation of real-time intrafraction monitoring for photon RT will be used as a basis to discuss the implementation of these methods for particle RT.

Continuous generation of volumetric images during stereotactic body radiation therapy using periodic kV imaging and an external respiratory surrogate

We present a technique for continuous generation of volumetric images during SBRT using periodic kV imaging and an external respiratory surrogate signal to drive a patient-specific PCA motion model. Using the on-board imager, kV radiographs are acquired every 3 s and used to fit the parameters of a motion model so that it matches observed changes in internal patient anatomy. A multi-dimensional correlation model is established between the motion model parameters and the external surrogate position and velocity, enabling volumetric image reconstruction between kV imaging time points. Performance of the algorithm was evaluated using 10 realistic eXtended CArdiac-Torso (XCAT) digital phantoms including 3D anatomical respiratory deformation programmed with 3D tumor positions measured with orthogonal kV imaging of implanted fiducial gold markers. The clinically measured ground truth 3D tumor positions provided a dataset with realistic breathing irregularities, and the combination of periodic on-board kV imaging with recorded external respiratory surrogate signal was used for correlation modeling to account for any changes in internal-external correlation. The three-dimensional tumor positions are reconstructed with an average root mean square error (RMSE) of 1.47 mm, and an average 95th percentile 3D positional error of 2.80 mm compared with the clinically measured ground truth 3D tumor positions. This technique enables continuous 3D anatomical image generation based on periodic kV imaging of internal anatomy without the additional dose of continuous kV imaging. The 3D anatomical images produced using this method can be used for treatment verification and delivered dose computation in the presence of irregular respiratory motion.

Assessment of 3D motion modeling performance for dose accumulation mapping on the MR-linac by simultaneous multislice MRI

Hybrid MR-linac systems enable intrafraction motion monitoring during radiation therapy. Since time-resolved 3D MRI is still challenging, various motion models have been developed that rely on time-resolved 2D imaging. Continuous validation of these models is important for accurate dose accumulation mapping. In this study we used 2D simultaneous multislice (SMS) imaging to improve the PCA-based motion modeling method developed previously (Stemkens et al 2016 Phys. Med. Biol. 61 5335-55). From the additional simultaneously acquired slices, several independent motion models could be generated, which allowed for an assessment of the sensitivity of the motion model to the location of the time-resolved 2D slices. Additionally, the best model could be chosen at every time-point, increasing the method's robustness. Imaging experiments were performed in six healthy volunteers using three simultaneous slices, which generated three independent models per volunteer. For each model the motion traces of the liver tip and both kidneys were estimated. We found that the location of the 2D slices influenced the model's error in five volunteers significantly with a p -value  <0.05, and that selecting the best model at every time-point can improve the method. This allows for more accurate and robust motion characterization in MR-guided radiotherapy.

Quadratic penalty method for intensity-based deformable image registration and 4DCT lung motion recovery

Intensity-based deformable image registration (DIR) requires minimizing an image dissimilarity metric. Imaged anatomy, such as bones and vasculature, as well as the resolution of the digital grid, can often cause discontinuities in the corresponding objective function. Consequently, the application of a gradient-based optimization algorithm requires a preprocessing image smoothing to ensure the existence of necessary image derivatives. Simple block matching (exhaustive search) methods do not require image derivative approximations, but their general effectiveness is often hindered by erroneous solutions (outliers). Block match methods are therefore often coupled with a statistical outlier detection method to improve results.

PURPOSE

The purpose of this work is to present a spatially accurate, intensity-based DIR optimization formulation that can be solved with a straightforward gradient-free quadratic penalty algorithm and is suitable for 4D thoracic computed tomography (4DCT) registration. Additionally, a novel regularization strategy based on the well-known leave-one-out robust statistical model cross-validation method is introduced.

METHODS

The proposed Quadratic Penalty DIR (QPDIR) method minimizes both an image dissimilarity term, which is separable with respect to individual voxel displacements, and a regularization term derived from the classical leave-one-out cross-validation statistical method. The resulting DIR problem lends itself to a quadratic penalty function optimization approach, where each subproblem can be solved by straightforward block coordinate descent iteration.

RESULTS

The spatial accuracy of the method was assessed using expert-determined landmarks on ten 4DCT datasets available on www.dir-lab.com. The QPDIR algorithm achieved average millimeter spatial errors between 0.69 (0.91) and 1.19 (1.26) on the ten test cases. On all ten 4DCT test cases, the QPDIR method produced spatial accuracies that are superior or equivalent to those produced by current state-of-the-art methods. Moreover, QPDIR achieved accuracies at the resolution of the landmark error assessment (i.e., the interobserver error) on six of the ten cases.

CONCLUSION

The QPDIR algorithm is based on a simple quadratic penalty function formulation and a regularization term inspired by leave-one-out cross validation. The formulation lends itself to a parallelizable, gradient-free, block coordinate descent numerical optimization method. Numerical results indicate that the method achieves a high spatial accuracy on 4DCT inhale/exhale phases.

MRI-guidance for motion management in external beam radiotherapy: current status and future challenges

High precision conformal radiotherapy requires sophisticated imaging techniques to aid in target localisation for planning and treatment, particularly when organ motion due to respiration is involved. X-ray based imaging is a well-established standard for radiotherapy treatments. Over the last few years, the ability of magnetic resonance imaging (MRI) to provide radiation-free images with high-resolution and superb soft tissue contrast has highlighted the potential of this imaging modality for radiotherapy treatment planning and motion management. In addition, these advantageous properties motivated several recent developments towards combined MRI radiation therapy treatment units, enabling in-room MRI-guidance and treatment adaptation. The aim of this review is to provide an overview of the state-of-the-art in MRI-based image guidance for organ motion management in external beam radiotherapy. Methodological aspects of MRI for organ motion management are reviewed and their application in treatment planning, in-room guidance and adaptive radiotherapy described. Finally, a roadmap for an optimal use of MRI-guidance is highlighted and future challenges are discussed.

Clinical implementations of 4D pencil beam scanned particle therapy: Report on the 4D treatment planning workshop 2016 and 2017

In 2016 and 2017, the 8th and 9th 4D treatment planning workshop took place in Groningen (the Netherlands) and Vienna (Austria), respectively. This annual workshop brings together international experts to discuss research, advances in clinical implementation as well as problems and challenges in 4D treatment planning, mainly in spot scanned proton therapy. In the last two years several aspects like treatment planning, beam delivery, Monte Carlo simulations, motion modeling and monitoring, QA phantoms as well as 4D imaging were thoroughly discussed. This report provides an overview of discussed topics, recent findings and literature review from the last two years. Its main focus is to highlight translation of 4D research into clinical practice and to discuss remaining challenges and pitfalls that still need to be addressed and to be overcome.

A 4D global respiratory motion model of the thorax based on CT images: A proof of concept

PURPOSE

Respiratory motion reduces the sensitivity and specificity of medical images especially in the thoracic and abdominal areas. It may affect applications such as cancer diagnostic imaging and/or radiation therapy (RT). Solutions to this issue include modeling of the respiratory motion in order to optimize both diagnostic and therapeutic protocols. Personalized motion modeling required patient-specific four-dimensional (4D) imaging which in the case of 4D computed tomography (4D CT) acquisition is associated with an increased dose. The goal of this work was to develop a global respiratory motion model capable of relating external patient surface motion to internal structure motion without the need for a patient-specific 4D CT acquisition.

METHODS

The proposed global model is based on principal component analysis and can be adjusted to a given patient anatomy using only one or two static CT images in conjunction with a respiratory synchronized patient external surface motion. It is based on the relation between the internal motion described using deformation fields obtained by registering 4D CT images and patient surface maps obtained either from optical imaging devices or extracted from CT image-based patient skin segmentation. 4D CT images of six patients were used to generate the global motion model which was validated by adapting it on four different patients having skin segmented surfaces and two other patients having time of flight camera acquired surfaces. The reproducibility of the proposed model was also assessed on two patients with two 4D CT series acquired within 2 weeks of each other.

RESULTS

Profile comparison shows the efficacy of the global respiratory motion model and an improvement while using two CT images in order to adapt the model. This was confirmed by the correlation coefficient with a mean correlation of 0.9 and 0.95 while using one or two CT images respectively and when comparing acquired to model generated 4D CT images. For the four patients with segmented surfaces, expert validation indicates an error of 2.35 ± 0.26 mm compared to 6.07 ± 0.76 mm when using a simple interpolation between full inspiration (FI) and full expiration (FE) CT only; i.e., without specific modeling of the respiratory motion. For the two patients with acquired surfaces, this error was of 2.48 ± 0.18 mm. In terms of reproducibility, model error changes of 0.12 and 0.17 mm were measured for the two patients concerned.

CONCLUSIONS

The framework for the derivation of a global respiratory motion model was developed. A single or two static CT images and associated patient surface motion, as a surrogate measure, are only needed to personalize the model. This model accuracy and reproducibility were assessed by comparing acquired vs model generated 4D CT images. Future work will consist of assessing extensively the proposed model for radiotherapy applications.

A prospective gating method to acquire a diverse set of free-breathing CT images for model-based 4DCT

Breathing motion modeling requires observation of tissues at sufficiently distinct respiratory states for proper 4D characterization. This work proposes a method to improve sampling of the breathing cycle with limited imaging dose. We designed and tested a prospective free-breathing acquisition protocol with a simulation using datasets from five patients imaged with a model-based 4DCT technique. Each dataset contained 25 free-breathing fast helical CT scans with simultaneous breathing surrogate measurements. Tissue displacements were measured using deformable image registration. A correspondence model related tissue displacement to the surrogate. Model residual was computed by comparing predicted displacements to image registration results. To determine a stopping criteria for the prospective protocol, i.e. when the breathing cycle had been sufficiently sampled, subsets of N scans where 5  ⩽  N  ⩽  9 were used to fit reduced models for each patient. A previously published metric was employed to describe the phase coverage, or 'spread', of the respiratory trajectories of each subset. Minimum phase coverage necessary to achieve mean model residual within 0.5 mm of the full 25-scan model was determined and used as the stopping criteria. Using the patient breathing traces, a prospective acquisition protocol was simulated. In all patients, phase coverage greater than the threshold necessary for model accuracy within 0.5 mm of the 25 scan model was achieved in six or fewer scans. The prospectively selected respiratory trajectories ranked in the (97.5  ±  4.2)th percentile among subsets of the originally sampled scans on average. Simulation results suggest that the proposed prospective method provides an effective means to sample the breathing cycle with limited free-breathing scans. One application of the method is to reduce the imaging dose of a previously published model-based 4DCT protocol to 25% of its original value while achieving mean model residual within 0.5 mm.