Multimodal medical volumetric data fusion using 3-D discrete shearlet transform and global-to-local rule

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

Multi-Dimensional Medical Image Fusion With Complex Sparse Representation

In the field of medical imaging, the fusion of data from diverse modalities plays a pivotal role in advancing our understanding of pathological conditions. Sparse representation (SR), a robust signal modeling technique, has demonstrated noteworthy success in multi-dimensional (MD) medical image fusion. However, a fundamental limitation appearing in existing SR models is their lack of directionality, restricting their efficacy in extracting anatomical details from different imaging modalities. To tackle this issue, we propose a novel directional SR model, termed complex sparse representation (ComSR), specifically designed for medical image fusion. ComSR independently represents MD signals over directional dictionaries along specific directions, allowing precise analysis of intricate details of MD signals. Besides, current studies in medical image fusion mostly concentrate on addressing either 2D or 3D fusion problems. This work bridges this gap by proposing a MD medical image fusion method based on ComSR, presenting a unified framework for both 2D and 3D fusion tasks. Experimental results across six multi-modal medical image fusion tasks, involving 93 pairs of 2D source images and 20 pairs of 3D source images, substantiate the superiority of our proposed method over 11 state-of-the-art 2D fusion methods and 4 representative 3D fusion methods, in terms of both visual quality and objective evaluation.

Brain tumor segmentation in multimodal MRI via pixel-level and feature-level image fusion

Brain tumor segmentation in multimodal MRI volumes is of great significance to disease diagnosis, treatment planning, survival prediction and other relevant tasks. However, most existing brain tumor segmentation methods fail to make sufficient use of multimodal information. The most common way is to simply stack the original multimodal images or their low-level features as the model input, and many methods treat each modality data with equal importance to a given segmentation target. In this paper, we introduce multimodal image fusion technique including both pixel-level fusion and feature-level fusion for brain tumor segmentation, aiming to achieve more sufficient and finer utilization of multimodal information. At the pixel level, we present a convolutional network named PIF-Net for 3D MR image fusion to enrich the input modalities of the segmentation model. The fused modalities can strengthen the association among different types of pathological information captured by multiple source modalities, leading to a modality enhancement effect. At the feature level, we design an attention-based modality selection feature fusion (MSFF) module for multimodal feature refinement to address the difference among multiple modalities for a given segmentation target. A two-stage brain tumor segmentation framework is accordingly proposed based on the above components and the popular V-Net model. Experiments are conducted on the BraTS 2019 and BraTS 2020 benchmarks. The results demonstrate that the proposed components on both pixel-level and feature-level fusion can effectively improve the segmentation accuracy of brain tumors.

Discrete Shearlets as a Sparsifying Transform in Low-Rank Plus Sparse Decomposition for Undersampled (k, t)-Space MR Data

The discrete shearlet transformation accurately represents the discontinuities and edges occurring in magnetic resonance imaging, providing an excellent option of a sparsifying transform. In the present paper, we examine the use of discrete shearlets over other sparsifying transforms in a low-rank plus sparse decomposition problem, denoted by L+S. The proposed algorithm is evaluated on simulated dynamic contrast enhanced (DCE) and small bowel data. For the small bowel, eight subjects were scanned; the sequence was run first on breath-holding and subsequently on free-breathing, without changing the anatomical position of the subject. The reconstruction performance of the proposed algorithm was evaluated against k-t FOCUSS. L+S decomposition, using discrete shearlets as sparsifying transforms, successfully separated the low-rank (background and periodic motion) from the sparse component (enhancement or bowel motility) for both DCE and small bowel data. Motion estimated from low-rank of DCE data is closer to ground truth deformations than motion estimated from L and S. Motility metrics derived from the S component of free-breathing data were not significantly different from the ones from breath-holding data up to four-fold undersampling, indicating that bowel (rapid/random) motility is isolated in S. Our work strongly supports the use of discrete shearlets as a sparsifying transform in a L+S decomposition for undersampled MR data.

FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.

A phase congruency-based green fluorescent protein and phase contrast image fusion method in nonsubsampled shearlet transform domain

Image fusion technique is an effective way to merge the information contained in different imaging modalities by generating a more informative composite image. Fusion of green fluorescent protein (GFP) and phase contrast images is of great significance to the subcellular localization, the functional analysis of protein, and the expression of gene. In this article, a phase congruency (PC)-based GFP and phase contrast image fusion method in nonsubsampled shearlet transform (NSST) domain is presented. The input images are decomposed by the NSST to acquire the multiscale and multidirection representations. The high-frequency coefficients are fused with a strategy based on PC and parameter-adaptive pulse coupled neural network (PA-PCNN), while the low-frequency coefficients are integrated through a local energy (LE)-based rule. Finally, the fused image is generated by conducting the inverse NSST on the merged high- and low-frequency coefficients. Experimental results illustrate that the presented method outperforms several state-of-the-art GFP and phase contrast image fusion algorithms on both qualitative and quantitative assessments.

Multi-Modality Medical Image Fusion Using Convolutional Neural Network and Contrast Pyramid

Medical image fusion techniques can fuse medical images from different morphologies to make the medical diagnosis more reliable and accurate, which play an increasingly important role in many clinical applications. To obtain a fused image with high visual quality and clear structure details, this paper proposes a convolutional neural network (CNN) based medical image fusion algorithm. The proposed algorithm uses the trained Siamese convolutional network to fuse the pixel activity information of source images to realize the generation of weight map. Meanwhile, a contrast pyramid is implemented to decompose the source image. According to different spatial frequency bands and a weighted fusion operator, source images are integrated. The results of comparative experiments show that the proposed fusion algorithm can effectively preserve the detailed structure information of source images and achieve good human visual effects.

Application of Image Fusion in Diagnosis and Treatment of Liver Cancer

With the accelerated development of medical imaging equipment and techniques, image fusion technology has been effectively applied for diagnosis, biopsy and radiofrequency ablation, especially for liver tumor. Tumor treatment relying on a single medical imaging modality might face challenges, due to the deep positioning of the lesions, operation history and the specific background conditions of the liver disease. Image fusion technology has been employed to address these challenges. Using the image fusion technology, one could obtain real-time anatomical imaging superimposed by functional images showing the same plane to facilitate the diagnosis and treatments of liver tumors. This paper presents a review of the key principles of image fusion technology, its application in tumor treatments, particularly in liver tumors, and concludes with a discussion of the limitations and prospects of the image fusion technology.

Fusion of 3-D medical image gradient domain based on detail-driven and directional structure tensor

BACKGROUND

Multi-modal medical image fusion plays a crucial role in many areas of modern medicine like diagnosis and therapy planning.

OBJECTIVE

Due to the factor that the structure tensor has the property of preserving the image geometry, we utilized it to construct the directional structure tensor and further proposed an improved 3-D medical image fusion method.

METHOD

The local entropy metrics were used to construct the gradient weights of different source images, and the eigenvectors of traditional structure tensor were combined with the second-order derivatives of image to construct the directional structure tensor. In addition, the guided filtering was employed to obtain detail components of the source images and construct a fused gradient field with the enhanced detail. Finally, the fusion image was generated by solving the functional minimization problem.

RESULTS AND CONCLUSION

Experimental results demonstrated that this new method is superior to the traditional structure tensor and multi-scale analysis in both visual effect and quantitative assessment.

A New Deep Learning Based Multi-Spectral Image Fusion Method

In this paper, we present a new effective infrared (IR) and visible (VIS) image fusion method by using a deep neural network. In our method, a Siamese convolutional neural network (CNN) is applied to automatically generate a weight map which represents the saliency of each pixel for a pair of source images. A CNN plays a role in automatic encoding an image into a feature domain for classification. By applying the proposed method, the key problems in image fusion, which are the activity level measurement and fusion rule design, can be figured out in one shot. The fusion is carried out through the multi-scale image decomposition based on wavelet transform, and the reconstruction result is more perceptual to a human visual system. In addition, the visual qualitative effectiveness of the proposed fusion method is evaluated by comparing pedestrian detection results with other methods, by using the YOLOv3 object detector using a public benchmark dataset. The experimental results show that our proposed method showed competitive results in terms of both quantitative assessment and visual quality.

Tensor Sparse Representation for 3-D Medical Image Fusion Using Weighted Average Rule

OBJECTIVE

The technique of fusing multimodal medical images into single image has a great impact on the clinical diagnosis. The previous works mostly concern the two-dimensional (2-D) image fusion performed on each slice individually, that may destroy the 3-D correlation across adjacent slices. To address this issue, this paper proposes a novel 3-D image fusion scheme based on Tensor Sparse Representation (TSR).

METHODS

First, each medical volume is arranged as a three-order tensor, and represented by TSR with learned dictionaries. Second, a novel "weighted average" rule, calculated from the tensor sparse coefficients using 3-D local-to-global strategy. The weights are then employed to combine the multimodal medical volumes through weighted average.

RESULTS

The visual and objective comparisons show that the proposed method is competitive to the existing methods on various medical volumes in different imaging modalities.

CONCLUSION

The TSR-based 3-D fusion approach with weighted average rule can preserve the 3-D structure of medical volume, and reduce the low contrast and artifacts in fused product.

SIGNIFICANCE

The designed weights offer the effective assigned weights and accurate salience levels measure, which can improve the performance of fusion approach.

Log-Gabor energy based multimodal medical image fusion in NSCT domain

Multimodal medical image fusion is a powerful tool in clinical applications such as noninvasive diagnosis, image-guided radiotherapy, and treatment planning. In this paper, a novel nonsubsampled Contourlet transform (NSCT) based method for multimodal medical image fusion is presented, which is approximately shift invariant and can effectively suppress the pseudo-Gibbs phenomena. The source medical images are initially transformed by NSCT followed by fusing low- and high-frequency components. The phase congruency that can provide a contrast and brightness-invariant representation is applied to fuse low-frequency coefficients, whereas the Log-Gabor energy that can efficiently determine the frequency coefficients from the clear and detail parts is employed to fuse the high-frequency coefficients. The proposed fusion method has been compared with the discrete wavelet transform (DWT), the fast discrete curvelet transform (FDCT), and the dual tree complex wavelet transform (DTCWT) based image fusion methods and other NSCT-based methods. Visually and quantitatively experimental results indicate that the proposed fusion method can obtain more effective and accurate fusion results of multimodal medical images than other algorithms. Further, the applicability of the proposed method has been testified by carrying out a clinical example on a woman affected with recurrent tumor images.