Deep learning-based lung image registration: A review

Robust Fast Inter-Bin Image Registration for Undersampled Coronary MRI Based on a Learned Motion Prior

OBJECTIVE

To propose a 3D nonrigid registration method that accurately estimates the 3D displacement field from artifact-corrupted Coronary Magnetic Resonance Angiography (CMRA) images.

METHODS

We developed a novel registration framework for registration of artifact-corrupted images based on a 3D U-Net initializer and a deep unrolling network. By leveraging a supervised learning framework with training labels estimated from fully-sampled images, the unrolling network learns a task-specific motion prior which reduces motion estimation biases caused by undersampling artifacts in the source images. We evaluated the proposed method, UNROLL, against an iterative Free-Form Deformation (FFD) registration method and a recently proposed Respiratory Motion Estimation network (RespME-net) for 6-fold (in-distribution) and 11-fold (out-of-distribution) accelerated CMRA.

RESULTS

Compared to the baseline methods, UNROLL improved both the accuracy of motion estimation and the quality of motion-compensated CMRA reconstruction at 6-fold acceleration. Furthermore, even at 11-fold acceleration, which was not included during training, UNROLL still generated more accurate displacement fields than the baseline methods. The computational time of UNROLL for the whole 3D volume was only 2 seconds.

CONCLUSION

By incorporating a learned respiratory motion prior, the proposed method achieves highly accurate motion estimation despite the large acceleration.

SIGNIFICANCE

This work introduces a fast and accurate method to estimate the displacement field from low-quality source images. It has the potential to significantly improve the quality of motion-compensated reconstruction for highly accelerated 3D CMRA.

Three-stage registration pipeline for dynamic lung field of chest X-ray images based on convolutional neural networks

Background

The anatomically constrained registration network (AC-RegNet), which yields anatomically plausible results, has emerged as the state-of-the-art registration architecture for chest X-ray (CXR) images. Nevertheless, accurate lung field registration results may be more favored and exciting than the registration results of the entire CXR images and hold promise for dynamic lung field analysis in clinical practice.

Objective

Based on the above, a registration model of the dynamic lung field of CXR images based on AC-RegNet and static CXR images is urgently developed to register these dynamic lung fields for clinical quantitative analysis.

Methods

This paper proposes a fully automatic three-stage registration pipeline for the dynamic lung field of CXR images. First, the dynamic lung field mask images are generated from a pre-trained standard lung field segmentation model with the dynamic CXR images. Then, a lung field abstraction model is designed to generate the dynamic lung field images based on the dynamic lung field mask images and their corresponding CXR images. Finally, we propose a three-step registration training method to train the AC-RegNet, obtaining the registration network of the dynamic lung field images (AC-RegNet_V3).

Results

The proposed AC-RegNet_V3 with the four basic segmentation networks achieve the mean dice similarity coefficient (DSC) of 0.991, 0.993, 0.993, and 0.993, mean Hausdorff distance (HD) of 12.512, 12.813, 12.449, and 13.661, mean average symmetric surface distance (ASSD) of 0.654, 0.550, 0.572, and 0.564, and mean squared distance (MSD) of 559.098, 577.797, 548.189, and 559.652, respectively. Besides, compared to the dynamic CXR images, the mean DSC of these four basic segmentation networks with AC-RegNet has been significantly improved by 7.2, 7.4, 7.4, and 7.4% (p-value < 0.0001). Meanwhile, the mean HD has been significantly improved by 8.994, 8.693, 9.057, and 7.845 (p-value < 0.0001). Similarly, the mean ASSD has significantly improved by 4.576, 4.680, 4.658, and 4.658 (p-value < 0.0001). Last, the mean MSD has significantly improved by 508.936, 519.776, 517.904, and 520.626 (p-value < 0.0001).

Conclusion

Our proposed three-stage registration pipeline has demonstrated its effectiveness in dynamic lung field registration. Therefore, it could become a powerful tool for dynamic lung field analysis in clinical practice, such as pulmonary airflow detection and air trapping location.

UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration

Complicated image registration is a key issue in medical image analysis, and deep learning-based methods have achieved better results than traditional methods. The methods include ConvNet-based and Transformer-based methods. Although ConvNets can effectively utilize local information to reduce redundancy via small neighborhood convolution, the limited receptive field results in the inability to capture global dependencies. Transformers can establish long-distance dependencies via a self-attention mechanism; however, the intense calculation of the relationships among all tokens leads to high redundancy. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network, which can enhance feature representation learning in the encoder and generate detailed displacement fields in the decoder to overcome these problems. We first propose a fusion attention block to integrate the advantages of ConvNets and Transformers, which inserts a ConvNet-based channel attention module into a multihead self-attention module. The overlapping attention block, a novel cross-attention method, uses overlapping windows to obtain abundant correlations with match information of a pair of images. Then, the blocks are flexibly stacked into a new powerful encoder. The decoder generation process of a high-resolution deformation displacement field from low-resolution features is considered as a superresolution process. Specifically, the superresolution module was employed to replace interpolation upsampling, which can overcome feature degradation. UTSRMorph was compared to state-of-the-art registration methods in the 3D brain MR (OASIS, IXI) and MR-CT datasets (abdomen, craniomaxillofacial). The qualitative and quantitative results indicate that UTSRMorph achieves relatively better performance. The code and datasets are publicly available at https://github.com/Runshi-Zhang/UTSRMorph.

GMmorph: dynamic spatial matching registration model for 3D medical image based on gated Mamba

Objective.Deformable registration aims to achieve nonlinear alignment of image space by estimating a dense displacement field. It is commonly used as a preprocessing step in clinical and image analysis applications, such as surgical planning, diagnostic assistance, and surgical navigation. We aim to overcome these challenges: Deep learning-based registration methods often struggle with complex displacements and lack effective interaction between global and local feature information. They also neglect the spatial position matching process, leading to insufficient registration accuracy and reduced robustness when handling abnormal tissues.Approach.We propose a dual-branch interactive registration model architecture from the perspective of spatial matching. Implicit regularization is achieved through a consistency loss, enabling the network to balance high accuracy with a low folding rate. We introduced the dynamic matching module between the two branches of the registration, which generates learnable offsets based on all the tokens across the entire resolution range of the base branch features. Using trilinear interpolation, the model adjusts its feature expression range according to the learned offsets, capturing highly flexible positional differences. To facilitate the spatial matching process, we designed the gated mamba layer to globally model pixel-level features by associating all voxel information, while the detail enhancement module, which is based on channel and spatial attention, enhances the richness of local feature details.Main results.Our study explores the model's performance in single-modal and multi-modal image registration, including normal brain, brain tumor, and lung images. We propose unsupervised and semi-supervised registration modes and conduct extensive validation experiments. The results demonstrate that the model achieves state-of-the-art performance across multiple datasets.Significance.By introducing a novel perspective of position matching, the model achieves precise registration of various types of medical data, offering significant clinical value in medical applications.

Detail-preserving image warping by enforcing smooth image sampling

Multi-phase dynamic contrast-enhanced magnetic resonance imaging image registration makes a substantial contribution to medical image analysis. However, existing methods (e.g., VoxelMorph, CycleMorph) often encounter the problem of image information misalignment in deformable registration tasks, posing challenges to the practical application. To address this issue, we propose a novel smooth image sampling method to align full organic information to realize detail-preserving image warping. In this paper, we clarify that the phenomenon about image information mismatch is attributed to imbalanced sampling. Then, a sampling frequency map constructed by sampling frequency estimators is utilized to instruct smooth sampling by reducing the spatial gradient and discrepancy between all-ones matrix and sampling frequency map. In addition, our estimator determines the sampling frequency of a grid voxel in the moving image by aggregating the sum of interpolation weights from warped non-grid sampling points in its vicinity and vectorially constructs sampling frequency map through projection and scatteration. We evaluate the effectiveness of our approach through experiments on two in-house datasets. The results showcase that our method preserves nearly complete details with ideal registration accuracy compared with several state-of-the-art registration methods. Additionally, our method exhibits a statistically significant difference in the regularity of the registration field compared to other methods, at a significance level of p < 0.05. Our code will be released at https://github.com/QingRui-Sha/SFM.

MSRA-Net: multi-channel semantic-aware and residual attention mechanism network for unsupervised 3D image registration

Objective. Convolutional neural network (CNN) is developing rapidly in the field of medical image registration, and the proposed U-Net further improves the precision of registration. However, this method may discard certain important information in the process of encoding and decoding steps, consequently leading to a decline in accuracy. To solve this problem, a multi-channel semantic-aware and residual attention mechanism network (MSRA-Net) is proposed in this paper.Approach. Our proposed network achieves efficient information aggregation by cleverly extracting the features of different channels. Firstly, a context-aware module (CAM) is designed to extract valuable contextual information. And the depth-wise separable convolution is employed in the CAM to alleviate the computational burden. Then, a new multi-channel semantic-aware module (MCSAM) is designed for more comprehensive fusion of up-sampling features. Additionally, the residual attention module is introduced in the up-sampling process to extract more semantic information and minimize information loss.Main results. This study utilizes Dice score, average symmetric surface distance and negative Jacobian determinant evaluation metrics to evaluate the influence of registration. The experimental results demonstrate that our proposed MSRA-Net has the highest accuracy compared to several state-of-the-art methods. Moreover, our network has demonstrated the highest Dice score across multiple datasets, thereby indicating that the superior generalization capabilities of our model.Significance. The proposed MSRA-Net offers a novel approach to improve medical image registration accuracy, with implications for various clinical applications. Our implementation is available athttps://github.com/shy922/MSRA-Net.

Developing a deep learning model for predicting ovarian cancer in Ovarian-Adnexal Reporting and Data System Ultrasound (O-RADS US) Category 4 lesions: A multicenter study

PURPOSE

To develop a deep learning (DL) model for differentiating between benign and malignant ovarian tumors of Ovarian-Adnexal Reporting and Data System Ultrasound (O-RADS US) Category 4 lesions, and validate its diagnostic performance.

METHODS

A retrospective analysis of 1619 US images obtained from three centers from December 2014 to March 2023. DeepLabV3 and YOLOv8 were jointly used to segment, classify, and detect ovarian tumors. Precision and recall and area under the receiver operating characteristic curve (AUC) were employed to assess the model performance.

RESULTS

A total of 519 patients (including 269 benign and 250 malignant masses) were enrolled in the study. The number of women included in the training, validation, and test cohorts was 426, 46, and 47, respectively. The detection models exhibited an average precision of 98.68% (95% CI: 0.95-0.99) for benign masses and 96.23% (95% CI: 0.92-0.98) for malignant masses. Moreover, in the training set, the AUC was 0.96 (95% CI: 0.94-0.97), whereas in the validation set, the AUC was 0.93(95% CI: 0.89-0.94) and 0.95 (95% CI: 0.91-0.96) in the test set. The sensitivity, specificity, accuracy, positive predictive value, and negative predictive values for the training set were 0.943,0.957,0.951,0.966, and 0.936, respectively, whereas those for the validation set were 0.905,0.935, 0.935,0.919, and 0.931, respectively. In addition, the sensitivity, specificity, accuracy, positive predictive value, and negative predictive value for the test set were 0.925, 0.955, 0.941, 0.956, and 0.927, respectively.

CONCLUSION

The constructed DL model exhibited high diagnostic performance in distinguishing benign and malignant ovarian tumors in O-RADS US category 4 lesions.

Physics-informed motion registration of lung parenchyma across static CT images

Lung injuries, such as ventilator-induced lung injury and radiation-induced lung injury, can lead to heterogeneous alterations in the biomechanical behavior of the lungs. While imaging methods, e.g., X-ray and static computed tomography (CT), can point to regional alterations in lung structure between healthy and diseased tissue, they fall short of delineating timewise kinematic variations between the former and the latter. Image registration has gained recent interest as a tool to estimate the displacement experienced by the lungs during respiration via regional deformation metrics such as volumetric expansion and distortion. However, successful image registration commonly relies on a temporal series of image stacks with small displacements in the lungs across succeeding image stacks, which remains limited in static imaging. In this study, we have presented a finite element (FE) method to estimate strains from static images acquired at the end-expiration (EE) and end-inspiration (EI) timepoints, i.e., images with a large deformation between the two distant timepoints. Physiologically realistic loads were applied to the geometry obtained at EE to deform this geometry to match the geometry obtained at EI. The results indicated that the simulation could minimize the error between the two geometries. Using four-dimensional (4D) dynamic CT in a rat, the strain at an isolated transverse plane estimated by our method showed sufficient agreement with that estimated through non-rigid image registration that used all the timepoints. Through the proposed method, we can estimate the lung deformation at any timepoint between EE and EI. The proposed method offers a tool to estimate timewise regional deformation in the lungs using only static images acquired at EE and EI.

DART: DEFORMABLE ANATOMY-AWARE REGISTRATION TOOLKIT FOR LUNG CT REGISTRATION WITH KEYPOINTS SUPERVISION

Spatially aligning two computed tomography (CT) scans of the lung using automated image registration techniques is a challenging task due to the deformable nature of the lung. However, existing deep-learning-based lung CT registration models are not trained with explicit anatomical knowledge. We propose the deformable anatomy-aware registration toolkit (DART), a masked autoencoder (MAE)-based approach, to improve the keypoint-supervised registration of lung CTs. Our method incorporates features from multiple decoders of networks trained to segment anatomical structures, including the lung, ribs, vertebrae, lobes, vessels, and airways, to ensure that the MAE learns relevant features corresponding to the anatomy of the lung. The pretrained weights of the transformer encoder and patch embeddings are then used as the initialization for the training of downstream registration. We compare DART to existing state-of-the-art registration models. Our experiments show that DART outperforms the baseline models (Voxelmorph, ViT-V-Net, and MAE-TransRNet) in terms of target registration error of both corrField-generated keypoints with 17%, 13%, and 9% relative improvement, respectively, and bounding box centers of nodules with 27%, 10%, and 4% relative improvement, respectively. Our implementation is available at https://github.com/yunzhengzhu/DART.

METnet: A novel deep learning model predicting MET dysregulation in non-small-cell lung cancer on computed tomography images

BACKGROUND

Mesenchymal epithelial transformation (MET) is a key molecular target for diagnosis and treatment of non-small cell lung cancer (NSCLC). The corresponding molecularly targeted therapeutics have been approved by Food and Drug Administration (FDA), achieving promising results. However, current detection of MET dysregulation requires biopsy and gene sequencing, which is invasive, time-consuming and difficult to obtain tumor samples.

METHODS

To address the above problems, we developed a noninvasive and convenient deep learning (DL) model based on Computed tomography (CT) imaging data for prediction of MET dysregulation. We introduced the unsupervised algorithm RK-net for automated image processing and utilized the MedSAM large model to achieve automated tissue segmentation. Based on the processed CT images, we developed a DL model (METnet). The model based on the grouped convolutional block. We evaluated the performance of the model over the internal test dataset using the area under the receiver operating characteristic curve (AUROC) and accuracy. We conducted subgroup analysis on the basis of clinical data of the lung cancer patients and compared the performance of the model in different subgroups.

RESULTS

The model demonstrated a good discriminative ability over the internal test dataset. The accuracy of METnet was 0.746 with an AUC value of 0.793 (95% CI 0.714-0.871). The subgroup analysis revealed that the model exhibited similar performance across different subgroups.

CONCLUSIONS

METnet realizes prediction of MET dysregulation in NSCLC, holding promise for guiding precise tumor diagnosis and treatment at the molecular level.

In-silico CT lung phantom generated from finite-element mesh

Several lung diseases lead to alterations in regional lung mechanics, including ventilator- and radiation-induced lung injuries. Such alterations can lead to localized underventilation of the affected areas, resulting in the overdistension of the surrounding healthy regions. Thus, there has been growing interest in quantifying the dynamics of the lung parenchyma using regional biomechanical markers. Image registration through dynamic imaging has emerged as a powerful tool to assess lung parenchyma's kinematic and deformation behaviors during respiration. However, the difficulty in validating the image registration estimation of lung deformation, primarily due to the lack of ground-truth deformation data, has limited its use in clinical settings. To address this barrier, we developed a method to convert a finite-element (FE) mesh of the lung into a phantom computed tomography (CT) image, advantageously possessing ground-truth information included in the FE model. The phantom CT images generated from the FE mesh replicated the geometry of the lung and large airways that were included in the FE model. Using spatial frequency response, we investigated the effect of " imaging parameters" such as voxel size (resolution) and proximity threshold values on image quality. A series of high-quality phantom images generated from the FE model simulating the respiratory cycle will allow for the validation and evaluation of image registration-based estimations of lung deformation. In addition, the present method could be used to generate synthetic data needed to train machine-learning models to estimate kinematic biomarkers from medical images that could serve as important diagnostic tools to assess heterogeneous lung injuries.