A New Multi-Atlas Registration Framework for Multimodal Pathological Images Using Conventional Monomodal Normal Atlases

Multimodal Brain Tumor Segmentation Boosted by Monomodal Normal Brain Images

Many deep learning based methods have been proposed for brain tumor segmentation. Most studies focus on deep network internal structure to improve the segmentation accuracy, while valuable external information, such as normal brain appearance, is often ignored. Inspired by the fact that radiologists often screen lesion regions with normal appearance as reference in mind, in this paper, we propose a novel deep framework for brain tumor segmentation, where normal brain images are adopted as reference to compare with tumor brain images in a learned feature space. In this way, features at tumor regions, i.e., tumor-related features, can be highlighted and enhanced for accurate tumor segmentation. It is known that routine tumor brain images are multimodal, while normal brain images are often monomodal. This causes the feature comparison a big issue, i.e., multimodal vs. monomodal. To this end, we present a new feature alignment module (FAM) to make the feature distribution of monomodal normal brain images consistent/inconsistent with multimodal tumor brain images at normal/tumor regions, making the feature comparison effective. Both public (BraTS2022) and in-house tumor brain image datasets are used to evaluate our framework. Experimental results demonstrate that for both datasets, our framework can effectively improve the segmentation accuracy and outperforms the state-of-the-art segmentation methods. Codes are available at https://github.com/hb-liu/Normal-Brain-Boost-Tumor-Segmentation.

Latent shape image learning via disentangled representation for cross-sequence image registration and segmentation

PURPOSE

Cross-sequence magnetic resonance image (MRI) registration and segmentation are two essential steps in a variety of medical image analysis tasks. And have attracted considerable research interest. However, they remain challenging due to domain shifts between different sequences. This study is aiming at proposing a novel method via disentangled representations, latent shape image learning (LSIL), for cross-sequence image registration and segmentation.

METHODS

Images from different sequences were firstly decomposed into a shared domain-invariant shape space and a domain-specific appearance space via an unsupervised image-to-image translation approach. A latent shape image learning model is then built on the disentangled shape representations to generate latent shape images. A series of experiments including cross-sequence image registration and segmentation were performed to qualitatively and quantitatively verify the validity of our method. Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95) were adopted as our evaluation metrics.

RESULTS

The performance of our method was evaluated based on 2 datasets total of 50 MRIs. The experimental results showed the superiority of the proposed framework over the state-of-the-art cross-sequence registration and segmentation approaches. The proposed method shows the mean DSCs of 0.711 and 0.867, respectively, in cross-sequence registration and segmentation.

CONCLUSION

We proposed a novel method based on representation disentangling to solve the cross-sequence registration and segmentation problem. Experimental results prove the feasibility and generalization of the generated latent shape images. The proposed method demonstrates significant potential for use in clinical environments of missing sequences. The source code is available at https://github.com/wujiong-hub/LSIL .

Cross-Modality Multi-Atlas Segmentation Using Deep Neural Networks

Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of registration or label fusion procedures. In this work, we design a novel cross-modality MAS framework, which uses available atlases from a certain modality to segment a target image from another modality. To boost the computational efficiency of the framework, both the image registration and label fusion are achieved by well-designed deep neural networks. For the atlas-to-target image registration, we propose a bi-directional registration network (BiRegNet), which can efficiently align images from different modalities. For the label fusion, we design a similarity estimation network (SimNet), which estimates the fusion weight of each atlas by measuring its similarity to the target image. SimNet can learn multi-scale information for similarity estimation to improve the performance of label fusion. The proposed framework was evaluated by the left ventricle and liver segmentation tasks on the MM-WHS and CHAOS datasets, respectively. Results have shown that the framework is effective for cross-modality MAS in both registration and label fusion. The code will be released publicly on https://github.com/NanYoMy/cmmas once the manuscript is accepted.

Multi-modality Large Deformation Diffeomorphic Metric Mapping Driven by Single-modality Images

Multi-modality magnetic resonance image (MRI) registration is an essential step in various MRI analysis tasks. However, it is challenging to have all required modalities in clinical practice, and thus the application of multi-modality registration is limited. This paper tackles such problem by proposing a novel unsupervised deep learning based multi-modality large deformation diffeomorphic metric mapping (LDDMM) framework which is capable of performing multi-modality registration only using single-modality MRIs. Specifically, an unsupervised image-to-image translation model is trained and used to synthesize the missing modality MRIs from the available ones. Multi-modality LDDMM is then performed in a multi-channel manner. Experimental results obtained on one publicly- accessible datasets confirm the superior performance of the proposed approach.Clinical relevance-This work provides a tool for multi-modality MRI registration with solely single-modality images, which addresses the very common issue of missing modalities in clinical practice.

A Disentangled Representations based Unsupervised Deformable Framework for Cross-modality Image Registration

Cross-modality magnetic resonance image (MRI) registration is a fundamental step in various MRI analysis tasks. However, it remains challenging due to the domain shift between different modalities. In this paper, we proposed a fully unsupervised deformable framework for cross-modality image registration through image disentangling. To be specific, MRIs of both modalities are decomposed into a shared domain-invariant content space and domain-specific style spaces via a multi-modal unsupervised image-to-image translation approach. An unsupervised deformable network is then built based on the assumption that intrinsic information in the content space is preserved among different modalities. In addition, we proposed a novel loss function consists of two metrics, with one defined in the original image space and the other in the content space. Validation experiments were performed on two datasets. Compared to two conventional state-of-the-art cross-modality registration methods, the proposed framework shows a superior registration performance.Clinical relevance-This work can serve as an auxiliary tool for cross-modality registration in clinical practice.

[A tissue recovery-based brain tumor image registration method]

We propose an algorithm for registration between brain tumor images and normal brain images based on tissue recovery. U-Net is first used in BraTS2018 dataset to segment the brain tumors, and PConv-Net is then used to simulate the generation of missing normal tissues in the tumor region to replace the tumor region. Finally, the normal brain image is registered to the tissue recovery brain image. We evaluated the effectiveness of this method by comparing the registration results of the repaired image and the tumor image corresponding to the surrounding tissues of the tumor area. The experimental results showed that the proposed method could reduce the effect of pathological variation, achieve a high registration accuracy, and effectively simulate and generate normal tissues to replace the tumor regions, thus improving the registration effect between brain tumor images and normal brain images.

A coarse-to-fine registration method for three-dimensional MR images

Three-dimensional (3D) multimodal magnetic resonance (MR) image registration aims to align similar things in different MR images spatially. Such a technology is useful in auxiliary disease diagnosis and surgical treatment. However, inconsistent intensity correspondence and large initial displacement contribute to the difficulty in registering multimodal MR volumes. A coarse-to-fine method is proposed in this study for pairwise 3D MR image rigid registration. Firstly, the proposed method extracts image feature points to form unregistered point sets and performs coarse registration based on point set registration to reduce the initial displacements of offset images effectively. Then, this method calculates a grey histogram based on voxels in the adaptive region of interest and further improves registration accuracy by maximizing mutual information of coarse-registered images. Some representative registration methods are compared on the basis of three MR image datasets to evaluate the performance of the proposed method. Experimental results show that the proposed method improved more in registration success rate and accuracy compared with conventional registration methods, especially when initial displacements are large.

An Indirect Multimodal Image Registration and Completion Method Guided by Image Synthesis

Multimodal registration is a challenging task due to the significant variations exhibited from images of different modalities. CT and MRI are two of the most commonly used medical images in clinical diagnosis, since MRI with multicontrast images, together with CT, can provide complementary auxiliary information. The deformable image registration between MRI and CT is essential to analyze the relationships among different modality images. Here, we proposed an indirect multimodal image registration method, i.e., sCT-guided multimodal image registration and problematic image completion method. In addition, we also designed a deep learning-based generative network, Conditional Auto-Encoder Generative Adversarial Network, called CAE-GAN, combining the idea of VAE and GAN under a conditional process to tackle the problem of synthetic CT (sCT) synthesis. Our main contributions in this work can be summarized into three aspects: (1) We designed a new generative network called CAE-GAN, which incorporates the advantages of two popular image synthesis methods, i.e., VAE and GAN, and produced high-quality synthetic images with limited training data. (2) We utilized the sCT generated from multicontrast MRI as an intermediary to transform multimodal MRI-CT registration into monomodal sCT-CT registration, which greatly reduces the registration difficulty. (3) Using normal CT as guidance and reference, we repaired the abnormal MRI while registering the MRI to the normal CT.

Adversarial learning for mono- or multi-modal registration

This paper introduces an unsupervised adversarial similarity network for image registration. Unlike existing deep learning registration methods, our approach can train a deformable registration network without the need of ground-truth deformations and specific similarity metrics. We connect a registration network and a discrimination network with a deformable transformation layer. The registration network is trained with the feedback from the discrimination network, which is designed to judge whether a pair of registered images are sufficiently similar. Using adversarial training, the registration network is trained to predict deformations that are accurate enough to fool the discrimination network. The proposed method is thus a general registration framework, which can be applied for both mono-modal and multi-modal image registration. Experiments on four brain MRI datasets and a multi-modal pelvic image dataset indicate that our method yields promising registration performance in accuracy, efficiency and generalizability compared with state-of-the-art registration methods, including those based on deep learning.

Deep feature descriptor based hierarchical dense matching for X-ray angiographic images

Backgroud and Objective: X-ray angiography, a powerful technique for blood vessel visualization, is widely used for interventional diagnosis of coronary artery disease because of its fast imaging speed and perspective inspection ability. Matching feature points in angiographic images is a considerably challenging task due to repetitive weak-textured regions.

METHODS

In this paper, we propose an angiographic image matching method based on the hierarchical dense matching framework, where a novel deep feature descriptor is designed to compute multilevel correlation maps. In particular, the deep feature descriptor is computed by a deep learning model specifically designed and trained for angiographic images, thereby making the correlation maps more distinctive for corresponding feature points in different angiographic images. Moreover, point correspondences are further hierarchically extracted from multilevel correlation maps with the highest similarity response(s), which is relatively robust and accurate. To overcome the problem regarding the lack of training samples, the convolutional neural network (designed for deep feature descriptor) is initially trained on samples from natural images and then fine-tuned on manually annotated angiographic images. Finally, a dense matching completion method, based on the distance between deep feature descriptors, is proposed to generate dense matches between images.

RESULTS

The proposed method has been evaluated on the number and accuracy of extracted matches and the performance of subtraction images. Experiments on a variety of angiographic images show promising matching accuracy, compared with state-of-the-art methods.

CONCLUSIONS

The proposed angiographic image matching method is shown to be accurate and effective for feature matching in angiographic images, and further achieves good performance in image subtraction.

BIRNet: Brain image registration using dual-supervised fully convolutional networks

In this paper, we propose a deep learning approach for image registration by predicting deformation from image appearance. Since obtaining ground-truth deformation fields for training can be challenging, we design a fully convolutional network that is subject to dual-guidance: (1) Ground-truth guidance using deformation fields obtained by an existing registration method; and (2) Image dissimilarity guidance using the difference between the images after registration. The latter guidance helps avoid overly relying on the supervision from the training deformation fields, which could be inaccurate. For effective training, we further improve the deep convolutional network with gap filling, hierarchical loss, and multi-source strategies. Experiments on a variety of datasets show promising registration accuracy and efficiency compared with state-of-the-art methods.