A hybrid patient-specific biomechanical model based image registration method for the motion estimation of lungs

Deep learning-based lung image registration: A review

Lung image registration can effectively describe the relative motion of lung tissues, thereby helping to solve series problems in clinical applications. Since the lungs are soft and fairly passive organs, they are influenced by respiration and heartbeat, resulting in discontinuity of lung motion and large deformation of anatomic features. This poses great challenges for accurate registration of lung image and its applications. The recent application of deep learning (DL) methods in the field of medical image registration has brought promising results. However, a versatile registration framework has not yet emerged due to diverse challenges of registration for different regions of interest (ROI). DL-based image registration methods used for other ROI cannot achieve satisfactory results in lungs. In addition, there are few review articles available on DL-based lung image registration. In this review, the development of conventional methods for lung image registration is briefly described and a more comprehensive survey of DL-based methods for lung image registration is illustrated. The DL-based methods are classified according to different supervision types, including fully-supervised, weakly-supervised and unsupervised. The contributions of researchers in addressing various challenges are described, as well as the limitations of these approaches. This review also presents a comprehensive statistical analysis of the cited papers in terms of evaluation metrics and loss functions. In addition, publicly available datasets for lung image registration are also summarized. Finally, the remaining challenges and potential trends in DL-based lung image registration are discussed.

Assessing four-dimensional CT stress maps derived from patient-specific biomechanical models of the lung with pulmonary function test data in lung cancer patients

OBJECTIVE

This study aims to retrospectively compare the stress map of the lung with pulmonary function test (PFT) results in lung cancer patients and to evaluate the potential of the stress map as an imaging biomarker for chronic obstructive pulmonary disease (COPD).

METHODS

25 lung cancer patients with pre-treatment four-dimensional CT (4DCT) and PFT data were retrospectively analysed. PFT metrics were used to diagnose obstructive lung disease. For each patient, forced expiratory volume in 1 s (FEV1 % predicted) and the ratio of FEV1 and forced vital capacity (FEV1/FVC) were recorded. 4DCT and biomechanical model-deformable image registration (BM-DIR) were used to obtain the lung stress map. The relationship between the mean of the total lung stress and PFT data was evaluated, and the COPD classification grade was also evaluated.

RESULTS

The mean values of the total lung stress and FEV1 % predicted showed a significant strong correlation [R = 0.833, (p < 0.001)]. The mean values and FEV1/FVC showed a significant strong correlation [R = 0.805, (p < 0.001)]. For the total lung stress, the area under the curve and the optimal cut-off value were 0.94 and 510.8 Pa for the classification of normal or abnormal lung function, respectively.

CONCLUSION

This study has demonstrated the potential of lung stress maps based on BM-DIR to accurately assess lung function by comparing them with PFT data.

ADVANCES IN KNOWLEDGE

The derivation of stress map directly from 4DCT is novel method. The BM-DIR-based lung stress map can provide an accurate assessment of lung function.

Optimization of mesh generation for geometric accuracy, robustness, and efficiency of biomechanical-model-based deformable image registration

BACKGROUND

Successful generation of biomechanical-model-based deformable image registration (BM-DIR) relies on user-defined parameters that dictate surface mesh quality. The trial-and-error process to determine the optimal parameters can be labor-intensive and hinder DIR efficiency and clinical workflow.

PURPOSE

To identify optimal parameters in surface mesh generation as boundary conditions for a BM-DIR in longitudinal liver and lung CT images to facilitate streamlined image registration processes.

METHODS

Contrast-enhanced CT images of 29 colorectal liver cancer patients and end-exhale four-dimensional CT images of 26 locally advanced non-small cell lung cancer patients were collected. Different combinations of parameters that determine the triangle mesh quality (voxel side length and triangle edge length) were investigated. The quality of DIRs generated using these parameters was evaluated with metrics for geometric accuracy, robustness, and efficiency. Metrics for geometric accuracy included target registration error (TRE) of internal vessel bifurcations, dice similar coefficient (DSC), mean distance to agreement (MDA), Hausdorff distance (HD) for organ contours, and number of vertices in the triangle mesh. American Association of Physicists in Medicine Task Group 132 was used to ensure parameters met TRE, DSC, MDA recommendations before the comparison among the parameters. Robustness was evaluated as the success rate of DIR generation, and efficiency was evaluated as the total time to generate boundary conditions and compute finite element analysis.

RESULTS

Voxel side length of 0.2 cm and triangle edge length of 3 were found to be the optimal parameters for both liver and lung, with success rate of 1.00 and 0.98 and average DIR computation time of 100 and 143 s, respectively. For this combination, the average TRE, DSC, MDA, and HD were 0.38-0.40, 0.96-0.97, 0.09-0.12, and 0.87-1.17 mm, respectively.

CONCLUSION

The optimal parameters were found for the analyzed patients. The decision-making process described in this study serves as a recommendation for BM-DIR algorithms to be used for liver and lung. These parameters can facilitate consistence in the evaluation of published studies and more widespread utilization of BM-DIR in clinical practice.

Thorax Dynamic Modeling and Biomechanical Analysis of Chest Breathing in Supine Lying Position

During respiration, the expansion and contraction of the chest and abdomen are coupled with each other, presenting a complex torso movement pattern. A finite element (FE) model of chest breathing based on the HUMOS2 human body model was developed. One-dimensional muscle units with active contraction functions were incorporated into the model based on Hill's active muscle model so as to generate muscle contraction forces that can change over time. The model was validated by comparing it to the surface displacement of the chest and abdomen during respiration. Then, the mechanism of the coupled motion of the chest and abdomen was analyzed. The analyses revealed that since the abdominal wall muscles are connected to the lower edge of the rib cage through tendons, the movement of the rib cage may cause the abdominal wall muscles to be stretched in both horizontal and vertical in a supine position. The anteroposterior and the right-left diameters of the chest will increase at inspiration, while the right-left diameter of the abdomen will decrease even though the anteroposterior diameter of the abdomen increases. The external intercostal muscles at different regions had different effects on the motion of the ribs during respiration. In particular, the external intercostal muscles at the lateral region had a larger effect on pump handle movement than bucket handle movement, and the external intercostal muscles at the dorsal region had a greater influence on bucket handle movement than pump handle movement.

Hierarchical anatomical structure-aware based thoracic CT images registration

Accurate thoracic CT image registration remains challenging due to complex joint deformations and different motion patterns in multiple organs/tissues during breathing. To combat this, we devise a hierarchical anatomical structure-aware based registration framework. It affords a coordination scheme necessary for constraining a general free-form deformation (FFD) during thoracic CT registration. The key is to integrate the deformations of different anatomical structures in a divide-and-conquer way. Specifically, a deformation ability-aware dissimilarity metric is proposed for complex joint deformations containing large-scale flexible deformation of the lung region, rigid displacement of the bone region, and small-scale flexible deformation of the rest region. Furthermore, a motion pattern-aware regularization is devised to handle different motion patterns, which contain sliding motion along the lung surface, almost no displacement of the spine and smooth deformation of other regions. Moreover, to accommodate large-scale deformation, a novel hierarchical strategy, wherein different anatomical structures are fused on the same control lattice, registers images from coarse to fine via elaborate Gaussian pyramids. Extensive experiments and comprehensive evaluations have been executed on the 4D-CT DIR and 3D DIR COPD datasets. It confirms that this newly proposed method is locally comparable to state-of-the-art registration methods specializing in local deformations, while guaranteeing overall accuracy. Additionally, in contrast to the current popular learning-based methods that typically require dozens of hours or more pre-training with powerful graphics cards, our method only takes an average of 63 s to register a case with an ordinary graphics card of RTX2080 SUPER, making our method still worth promoting. Our code is available at https://github.com/heluxixue/Structure_Aware_Registration/tree/master.

Projection-based dynamic tomography

Objective. This paper proposes a 4D dynamic tomography framework that allows a moving sample to be imaged in a tomograph as well as the associated space-time kinematics to be measured with nothing more than a single conventional x-ray scan.Approach. The method exploits the consistency of the projection/reconstruction operations through a multi-scale procedure. The procedure is composed of two main parts solved alternatively: a motion-compensated reconstruction algorithm and a projection-based measurement procedure that estimates the displacement field directly on each projection.Main results. The method is applied to two studies: a numerical simulation of breathing from chest computed tomography (4D-CT) and a clinical cone-beam CT of a breathing patient acquired for image guidance of radiotherapy. The reconstructed volume, initially blurred by the motion, is cleaned from motion artifacts.Significance. Applying the proposed approach results in an improved reconstruction quality showing sharper edges and finer details.

Mechanical properties of whole-body soft human tissues: a review

The mechanical properties of soft tissues play a key role in studying human injuries and their mitigation strategies. While such properties are indispensable for computational modelling of biological systems, they serve as important references in loading and failure experiments, and also for the development of tissue simulants. To date, experimental studies have measured the mechanical properties of peripheral tissues (e.g. skin)in-vivoand limited internal tissuesex-vivoin cadavers (e.g. brain and the heart). The lack of knowledge on a majority of human tissues inhibit their study for applications ranging from surgical planning, ballistic testing, implantable medical device development, and the assessment of traumatic injuries. The purpose of this work is to overcome such challenges through an extensive review of the literature reporting the mechanical properties of whole-body soft tissues from head to toe. Specifically, the available linear mechanical properties of all human tissues were compiled. Non-linear biomechanical models were also introduced, and the soft human tissues characterized using such models were summarized. The literature gaps identified from this work will help future biomechanical studies on soft human tissue characterization and the development of accurate medical models for the study and mitigation of injuries.

Validation of a CT-based motion model with in-situ fluoroscopy for lung surface deformation estimation

Many surrogate-based motion models (SMMs), proposed to guide motion management in radiotherapy, are constructed by correlating motion of an external surrogate and internal anatomy during CT-simulation. Changes in this correlation define model break down. We validate a methodology that incorporates fluoroscopic images (FL) acquired during treatment for SMM construction and update. Under a prospective IRB, 4DCT scans, VisionRT surfaces, and orthogonal FLs were collected from five lung cancer patients. VisionRT surfaces and two FL time-series were acquired pre- and post-treatment. A simulated annealing optimization scheme was used to estimate optimal lung deformations by maximizing the mutual information between digitally reconstructed radiographs (DRRs) of the SMM-estimated 3D images and FLs. Our SMM used partial-least-regression and was trained using the optimal deformations and VisionRT surfaces from the first breathing-cycle. SMM performance was evaluated using the mutual information score between reference FLs and the corresponding SMM or phase-assigned 4DCT DRRs. The Hausdorff distance for contoured landmarks was used to evaluate target position estimation error. For four out of five patients, two principal components approximated lung surface deformations with submillimeter accuracy. Analysis of the mutual information score between more than 4,000 pairs of FL and DRR demonstrated that our model led to more similarity between the FL and DRR images compared to 4DCT and DRR images from a model based on an a priori correlation model. Our SMM consistently displayed lower mean and 95^th percentile Hausdorff distances. For one patient, 95^th percentile Hausdorff distance was reduced by 11mm. Patient-averaged reductions in mean and 95^th percentile Hausdorff distances were 3.6mm and 7mm for right-lung, and 3.1mm and 4mm for left-lung targets. FL data were used to evaluate model performance and investigate the feasibility of model update. Despite variability in breathing, use of post-treatment FL preserved model fidelity and consistently outperformed 4DCT for position estimation.

A hybrid, image-based and biomechanics-based registration approach to markerless intraoperative nodule localization during video-assisted thoracoscopic surgery

The resection of small, low-dense or deep lung nodules during video-assisted thoracoscopic surgery (VATS) is surgically challenging. Nodule localization methods in clinical practice typically rely on the preoperative placement of markers, which may lead to clinical complications. We propose a markerless lung nodule localization framework for VATS based on a hybrid method combining intraoperative cone-beam CT (CBCT) imaging, free-form deformation image registration, and a poroelastic lung model with allowance for air evacuation. The difficult problem of estimating intraoperative lung deformations is decomposed into two more tractable sub-problems: (i) estimating the deformation due the change of patient pose from preoperative CT (supine) to intraoperative CBCT (lateral decubitus); and (ii) estimating the pneumothorax deformation, i.e. a collapse of the lung within the thoracic cage. We were able to demonstrate the feasibility of our localization framework with a retrospective validation study on 5 VATS clinical cases. Average initial errors in the range of 22 to 38 mm were reduced to the range of 4 to 14 mm, corresponding to an error correction in the range of 63 to 85%. To our knowledge, this is the first markerless lung deformation compensation method dedicated to VATS and validated on actual clinical data.

Towards Non-Invasive Lung Tumor Tracking Based on Patient Specific Model of Respiratory System

The goal of this paper is to calculate a complex internal respiratory and tumoral movements by measuring respiratory air flows and thorax movements. In this context, we present a new lung tumor tracking approach based on a patient-specific biomechanical model of the respiratory system, which takes into account the physiology of respiratory motion to simulate the real non-reproducible motion. The behavior of the lungs, is directly driven by the simulated actions of the breathing muscles, i.e. the diaphragm and the intercostal muscles (the rib cage). In this paper, the lung model is monitored and controlled by a personalized lung pressure/volume relationship during a whole respiratory cycle. The lung pressure and rib kinematics are patient specific and obtained by surrogate measurement. The rib displacement corresponding to the transformation which was computed by finite helical axis method from the end of exhalation (EE) to the end of inhalation (EI). The lung pressure is calculated by an optimization framework based on inverse finite element analysis, by minimizing the lung volume errors, between the respiratory volume (respiratory airflow exchange) and the simulated volume (calculated by biomechanical simulation). We have evaluated the model accuracy on five public datasets. We have also evaluated the lung tumor motion identified in 4D CT scan images and compared it with the trajectory that was obtained by finite element simulation. The effects of rib kinematics on lung tumor trajectory were investigated. Over all phases of respiration, our developed model is able to predict the lung tumor motion with an average landmark error of [Formula: see text]. The results demonstrate the effectiveness of our physics-based model. We believe that this model can be potentially used in 4D dose computation, removal of breathing motion artifacts in positron emission tomography (PET) or gamma prompt image reconstruction.

In-vivo lung biomechanical modeling for effective tumor motion tracking in external beam radiation therapy

Lung cancer is the most common cause of cancer-related death in both men and women. Radiation therapy is widely used for lung cancer treatment; however, respiratory motion presents challenges that can compromise the accuracy and/or effectiveness of radiation treatment. Respiratory motion compensation using biomechanical modeling is a common approach used to address this challenge. This study focuses on the development and validation of a lung biomechanical model that can accurately estimate the motion and deformation of lung tumor. Towards this goal, treatment planning 4D-CT images of lung cancer patients were processed to develop patient-specific finite element (FE) models of the lung to predict the patients' tumor motion/deformation. The tumor motion/deformation was modeled for a full respiration cycle, as captured by the 4D-CT scans. Parameters driving the lung and tumor deformation model were found through an inverse problem formulation. The CT datasets pertaining to the inhalation phases of respiration were used for validating the model's accuracy. The volumetric Dice similarity coefficient between the actual and simulated gross tumor volumes (GTVs) of the patients calculated across respiration phases was found to range between 0.80 ± 0.03 and 0.92 ± 0.01. The average error in estimating tumor's center of mass calculated across respiration phases ranged between 0.50 ± 0.10 (mm) and 1.04 ± 0.57 (mm), indicating a reasonably good accuracy of the proposed model. The proposed model demonstrates favorable accuracy for estimating the lung tumor motion/deformation, and therefore can potentially be used in radiation therapy applications for respiratory motion compensation.

Generation of a local lung respiratory motion model using a weighted sparse algorithm and motion prior-based registration

Respiration-introduced tumor location uncertainty is a challenge in lung percutaneous interventions, especially for the respiratory motion estimation of the tumor and surrounding vessel structures. In this work, a local motion modeling method is proposed based on whole-chest computed tomography (CT) and CT-fluoroscopy (CTF) scans. A weighted sparse statistical modeling (WSSM) method that can accurately capture location errors for each landmark point is proposed for lung motion prediction. By varying the sparse weight coefficients of the WSSM method, newly input motion information is approximately represented by a sparse linear combination of the respiratory motion repository and employed to serve as prior knowledge for the following registration process. We have also proposed an adaptive motion prior-based registration method to improve the motion prediction accuracy of the motion model in the region of interest (ROI). This registration method adopts a B-spline scheme to interactively weight the relative influence of the prior knowledge, model surface and image intensity information by locally controlling the deformation in the CTF image region. The proposed method has been evaluated on 15 image pairs between the end-expiratory (EE) and end-inspiratory (EI) phases and 31 four-dimensional CT (4DCT) datasets. The results reveal that the proposed WSSM method achieved a better motion prediction performance than other existing lung statistical motion modeling methods, and the motion prior-based registration method can generate more accurate local motion information in the ROI.

Incorporating Pathology-Induced Heterogeneities in a Patient-Specific Biomechanical Model of the Lung for Accurate Tumor Motion Estimation

Radiation therapy (RT) is an important component of treatment for lung cancer. However, the accuracy of this method can be affected by the complex respiratory motion/deformation of the target tumor during treatment. To improve the accuracy of RT, patient-specific biomechanical models of the lung have been proposed for estimating the tumor's respiratory motion/deformation. Chronic obstructive pulmonary disease (COPD) has a high incidence among lung cancer patients and is associated with heterogeneous destruction of lung parenchyma. This key heterogeneity element, however, has not been incorporated in lung biomechanical models developed in previous studies. In this work, we have developed a physiologically and patho-physiologically realistic lung biomechanical model that accounts for lung tissue heterogeneity. Four-dimensional computed tomography (4DCT) images were used to build a patient-specific finite element (FE) model of the lung. Image information was used to identify and incorporate inhomogeneities within the model. Mechanical properties of normal and diseased regions in the lung and the transpulmonary pressure driving the respiratory motion were estimated using an optimization algorithm that maximizes the similarity between the actual and simulated tumor and lung image data. Results from this proof of concept study on a lung cancer patient indicated improved accuracy of tumor motion estimation when COPD-induced lung tissue heterogeneities were incorporated in the model.

LungRegNet: An unsupervised deformable image registration method for 4D-CT lung

PURPOSE

To develop an accurate and fast deformable image registration (DIR) method for four-dimensional computed tomography (4D-CT) lung images. Deep learning-based methods have the potential to quickly predict the deformation vector field (DVF) in a few forward predictions. We have developed an unsupervised deep learning method for 4D-CT lung DIR with excellent performances in terms of registration accuracies, robustness, and computational speed.

METHODS

A fast and accurate 4D-CT lung DIR method, namely LungRegNet, was proposed using deep learning. LungRegNet consists of two subnetworks which are CoarseNet and FineNet. As the name suggests, CoarseNet predicts large lung motion on a coarse scale image while FineNet predicts local lung motion on a fine scale image. Both the CoarseNet and FineNet include a generator and a discriminator. The generator was trained to directly predict the DVF to deform the moving image. The discriminator was trained to distinguish the deformed images from the original images. CoarseNet was first trained to deform the moving images. The deformed images were then used by the FineNet for FineNet training. To increase the registration accuracy of the LungRegNet, we generated vessel-enhanced images by generating pulmonary vasculature probability maps prior to the network prediction.

RESULTS

We performed fivefold cross validation on ten 4D-CT datasets from our department. To compare with other methods, we also tested our method using separate 10 DIRLAB datasets that provide 300 manual landmark pairs per case for target registration error (TRE) calculation. Our results suggested that LungRegNet has achieved better registration accuracy in terms of TRE than other deep learning-based methods available in the literature on DIRLAB datasets. Compared to conventional DIR methods, LungRegNet could generate comparable registration accuracy with TRE smaller than 2 mm. The integration of both the discriminator and pulmonary vessel enhancements into the network was crucial to obtain high registration accuracy for 4D-CT lung DIR. The mean and standard deviation of TRE were 1.00 ± 0.53 mm and 1.59 ± 1.58 mm on our datasets and DIRLAB datasets respectively.

CONCLUSIONS

An unsupervised deep learning-based method has been developed to rapidly and accurately register 4D-CT lung images. LungRegNet has outperformed its deep-learning-based peers and achieved excellent registration accuracy in terms of TRE.

Locally Adaptive Total p-Variation Regularization for Non-Rigid Image Registration With Sliding Motion

Due to the complicated thoracic movements which contain both sliding motion occurring at lung surfaces and smooth motion within individual organs, respiratory estimation is still an intrinsically challenging task. In this paper, we propose a novel regularization term called locally adaptive total p-variation (LaTpV) and embed it into a parametric registration framework to accurately recover lung motion. LaTpV originates from a modified L_p-norm constraint (1 < p < 2), where a prior distribution of p modeled by the Dirac-shaped function is constructed to specifically assign different values to voxels. LaTpV adaptively balances the smoothness and discontinuity of the displacement field to encourage an expected sliding interface. Additionally, we also analytically deduce the gradient of the cost function with respect to transformation parameters. To validate the performance of LaTpV, we not only test it on two mono-modal databases including synthetic images and pulmonary computed tomography (CT) images, but also on a more difficult thoracic CT and positron emission tomography (PET) dataset for the first time. For all experiments, both the quantitative and qualitative results indicate that LaTpV significantly surpasses some existing regularizers such as bending energy and parametric total variation. The proposed LaTpV based registration scheme might be more superior for sliding motion correction and more potential for clinical applications such as the diagnosis of pleural mesothelioma and the adjustment of radiotherapy plans.

Use of Elastic Registration in Pulmonary MRI for the Assessment of Pulmonary Fibrosis in Patients with Systemic Sclerosis

Background Current imaging methods are not sensitive to changes in pulmonary function resulting from fibrosis. MRI with ultrashort echo time can be used to image the lung parenchyma and lung motion. Purpose To evaluate elastic registration of inspiratory-to-expiratory lung MRI for the assessment of pulmonary fibrosis in study participants with systemic sclerosis (SSc). Materials and Methods This prospective study was performed from September 2017 to March 2018 and recruited healthy volunteers and participants with SSc and high-resolution CT (within the previous 3 months) of the chest for lung MRI. Two breath-hold, coronal, three-dimensional, ultrashort-echo-time, gradient-echo sequences of the lungs were acquired after full inspiration and expiration with a 3.0-T unit. Images were registered from inspiration to expiration by using an elastic registration algorithm. Jacobian determinants were calculated from deformation fields and represented on color maps. Similarity between areas with marked shrinkage and logarithm of Jacobian determinants less than -0.15 were compared between healthy volunteers and study participants with SSc. Receiver operating characteristic curve analysis was performed to determine the best Dice similarity coefficient threshold for diagnosis of fibrosis. Results Sixteen participants with SSc (seven with pulmonary fibrosis at high-resolution CT) and 11 healthy volunteers were evaluated. Areas of marked shrinkage during expiration with logarithm of Jacobian determinants less than -0.15 were found in the posterior lung bases of healthy volunteers and in participants with SSc without fibrosis, but not in participants with fibrosis. The sensitivity and specificity of MRI for presence of fibrosis at high-resolution CT were 86% and 75%, respectively (area under the curve, 0.81; P = .04) by using a threshold of 0.36 for Dice similarity coefficient. Conclusion Elastic registration of inspiratory-to-expiratory MRI shows less lung base respiratory deformation in study participants with systemic sclerosis-related pulmonary fibrosis compared with participants without fibrosis. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Biederer in this issue.