Novel-view X-ray projection synthesis through geometry-integrated deep learning

PRSCS-Net: Progressive 3D/2D rigid Registration network with the guidance of Single-view Cycle Synthesis

The 3D/2D registration for 3D pre-operative images (computed tomography, CT) and 2D intra-operative images (X-ray) plays an important role in image-guided spine surgeries. Conventional iterative-based approaches suffer from time-consuming processes. Existing learning-based approaches require high computational costs and face poor performance on large misalignment because of projection-induced losses or ill-posed reconstruction. In this paper, we propose a Progressive 3D/2D rigid Registration network with the guidance of Single-view Cycle Synthesis, named PRSCS-Net. Specifically, we first introduce the differentiable backward/forward projection operator into the single-view cycle synthesis network, which reconstructs corresponding 3D geometry features from two 2D intra-operative view images (one from the input, and the other from the synthesis). In this way, the problem of limited views during reconstruction can be solved. Subsequently, we employ a self-reconstruction path to extract latent representation from pre-operative 3D CT images. The following pose estimation process will be performed in the 3D geometry feature space, which can solve the dimensional gap, greatly reduce the computational complexity, and ensure that the features extracted from pre-operative and intra-operative images are as relevant as possible to pose estimation. Furthermore, to enhance the ability of our model for handling large misalignment, we develop a progressive registration path, including two sub-registration networks, aiming to estimate the pose parameters via two-step warping volume features. Finally, our proposed method has been evaluated on a public dataset CTSpine1k and an in-house dataset C-ArmLSpine for 3D/2D registration. Results demonstrate that PRSCS-Net achieves state-of-the-art registration performance in terms of registration accuracy, robustness, and generalizability compared with existing methods. Thus, PRSCS-Net has potential for clinical spinal disease surgical planning and surgical navigation systems.

NuTracker: a coordinate-based neural network representation of lung motion for intrafraction tumor tracking with various surrogates in radiotherapy

Objective. Tracking tumors and surrounding tissues in real-time is critical for reducing errors and uncertainties during radiotherapy. Existing methods are either limited by the linear representation or scale poorly with the volume resolution. To address both issues, we propose a novel coordinate-based neural network representation of lung motion to predict the instantaneous 3D volume at arbitrary spatial resolution from various surrogates: patient surface, fiducial marker, and single kV projection.Approach. The proposed model, namely NuTracker, decomposes the 4DCT into a template volume and dense displacement fields (DDFs), and uses two coordinate neural networks to predict them from spatial coordinates and surrogate states. The predicted template is spatially warped with the predicted DDF to produce the deformed volume for a given surrogate state. The nonlinear coordinate networks enable representing complex motion at infinite resolution. The decomposition allows imposing different regularizations on the spatial and temporal domains. The meta-learning and multi-task learning are used to train NuTracker across patients and tasks, so that commonalities and differences can be exploited. NuTracker was evaluated on seven patients implanted with markers using a leave-one-phase-out procedure.Main results. The 3D marker localization error is 0.66 mm on average and <1 mm at 95th-percentile, which is about 26% and 32% improvement over the predominant linear methods. The tumor coverage and image quality are improved by 5.7% and 11% in terms of dice and PSNR. The difference in the localization error for different surrogates is small and is not statistically significant. Cross-population learning and multi-task learning contribute to performance. The model tolerates surrogate drift to a certain extent.Significance. NuTracker can provide accurate estimation for entire tumor volume based on various surrogates at infinite resolution. It is of great potential to apply the coordinate network to other imaging modalities, e.g. 4DCBCT and other tasks, e.g. 4D dose calculation.

NeRP: Implicit Neural Representation Learning With Prior Embedding for Sparsely Sampled Image Reconstruction

Image reconstruction is an inverse problem that solves for a computational image based on sampled sensor measurement. Sparsely sampled image reconstruction poses additional challenges due to limited measurements. In this work, we propose a methodology of implicit Neural Representation learning with Prior embedding (NeRP) to reconstruct a computational image from sparsely sampled measurements. The method differs fundamentally from previous deep learning-based image reconstruction approaches in that NeRP exploits the internal information in an image prior and the physics of the sparsely sampled measurements to produce a representation of the unknown subject. No large-scale data is required to train the NeRP except for a prior image and sparsely sampled measurements. In addition, we demonstrate that NeRP is a general methodology that generalizes to different imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). We also show that NeRP can robustly capture the subtle yet significant image changes required for assessing tumor progression.