Optimized dual-tree complex wavelet transform aided multimodal image fusion with adaptive weighted average fusion strategy

Image fusion is generally utilized for retrieving significant data from a set of input images to provide useful informative data. Image fusion enhances the applicability and quality of data. Hence, the analysis of multimodal image fusion is a new to the research topic, which is designed by combining the images of multimodal into single image in order to preserveexact details. On the other hand, the existing approaches face challenges in the precise interpretation of source images, and also it have only captured local information without considering the wide range of information. To consider these weaknesses, a multimodal image fusion model is planned to develop according to the multi-resolution transform along with the optimization strategy. At first, the images are effectively analyzed from standard public datasets and further, the images given into the Optimized Dual-Tree Complex Wavelet Transform (ODTCWT) to acquire low frequency and high frequency coefficients. Here, certain parameters in DTCWT get tuned with the hybridized heuristic strategy with the Probability of Fitness-based Honey Badger Squirrel Search Optimization (PF-HBSSO) to enhance the decomposition quality. Then, the fusion of high-frequency coefficients is performed using adaptive weighted average fusion technique, whereas the weights are optimized using PF-HBSSOto achieve the optimal fused results. Similarly, the low-frequency coefficients are combined by average fusion. Finally, the fused images undergo image reconstruction using the inverse ODTCWT. The experimental evaluation of the designed multimodal image fusion illustratessuperioritythat distinguishes this work from others.

A Review of EEG-based Localization of Epileptic Seizure Foci: Common Points with Multimodal Fusion of Brain Data

Unexpected seizures significantly decrease the quality of life in epileptic patients. Seizure attacks are caused by hyperexcitability and anatomical lesions of special regions of the brain, and cognitive impairments and memory deficits are their most common concomitant effects. In addition to seizure reduction treatments, medical rehabilitation involving brain-computer interfaces and neurofeedback can improve cognition and quality of life in patients with focal epilepsy in most cases, in particular when resective epilepsy surgery has been considered treatment in drug-resistant epilepsy. Source estimation and precise localization of epileptic foci can improve such rehabilitation and treatment. Electroencephalography (EEG) monitoring and multimodal noninvasive neuroimaging techniques such as ictal/interictal single-photon emission computerized tomography (SPECT) imaging and structural magnetic resonance imaging are common practices for the localization of epileptic foci and have been studied in several kinds of researches. In this article, we review the most recent research on EEG-based localization of seizure foci and discuss various methods, their advantages, limitations, and challenges with a focus on model-based data processing and machine learning algorithms. In addition, we survey whether combined analysis of EEG monitoring and neuroimaging techniques, which is known as multimodal brain data fusion, can potentially increase the precision of the seizure foci localization. To this end, we further review and summarize the key parameters and challenges of processing, fusion, and analysis of multiple source data, in the framework of model-based signal processing, for the development of a multimodal brain data analyzing system. This article has the potential to be used as a valuable resource for neuroscience researchers for the development of EEG-based rehabilitation systems based on multimodal data analysis related to focal epilepsy.

SegCoFusion: An Integrative Multimodal Volumetric Segmentation Cooperating With Fusion Pipeline to Enhance Lesion Awareness

Multimodal volumetric segmentation and fusion are two valuable techniques for surgical treatment planning, image-guided interventions, tumor growth detection, radiotherapy map generation, etc. In recent years, deep learning has demonstrated its excellent capability in both of the above tasks, while these methods inevitably face bottlenecks. On the one hand, recent segmentation studies, especially the U-Net-style series, have reached the performance ceiling in segmentation tasks. On the other hand, it is almost impossible to capture the ground truth of the fusion in multimodal imaging, due to differences in physical principles among imaging modalities. Hence, most of the existing studies in the field of multimodal medical image fusion, which fuse only two modalities at a time with hand-crafted proportions, are subjective and task-specific. To address the above concerns, this work proposes an integration of multimodal segmentation and fusion, namely SegCoFusion, which consists of a novel feature frequency dividing network named FDNet and a segmentation part using a dual-single path feature supplementing strategy to optimize the segmentation inputs and suture with the fusion part. Furthermore, focusing on multimodal brain tumor volumetric fusion and segmentation, the qualitative and quantitative results demonstrate that SegCoFusion can break the ceiling both of segmentation and fusion methods. Moreover, the effectiveness of the proposed framework is also revealed by comparing it with state-of-the-art fusion methods on 2D two-modality fusion tasks, our method achieves better fusion performance than others. Therefore, the proposed SegCoFusion develops a novel perspective that improves the performance in volumetric fusion by cooperating with segmentation and enhances lesion awareness.

An Improved Hybrid Network With a Transformer Module for Medical Image Fusion

Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.

A Non-Conventional Review on Multi-Modality-Based Medical Image Fusion

Today, medical images play a crucial role in obtaining relevant medical information for clinical purposes. However, the quality of medical images must be analyzed and improved. Various factors affect the quality of medical images at the time of medical image reconstruction. To obtain the most clinically relevant information, multi-modality-based image fusion is beneficial. Nevertheless, numerous multi-modality-based image fusion techniques are present in the literature. Each method has its assumptions, merits, and barriers. This paper critically analyses some sizable non-conventional work within multi-modality-based image fusion. Often, researchers seek help in apprehending multi-modality-based image fusion and choosing an appropriate multi-modality-based image fusion approach; this is unique to their cause. Hence, this paper briefly introduces multi-modality-based image fusion and non-conventional methods of multi-modality-based image fusion. This paper also signifies the merits and downsides of multi-modality-based image fusion.

T 1-T 2 dual-modal magnetic resonance contrast-enhanced imaging for rat liver fibrosis stage

The development of an effective method for staging liver fibrosis has always been a hot topic of research in the field of liver fibrosis. In this paper, PEGylated ultrafine superparamagnetic iron oxide nanocrystals (SPIO@PEG) were developed for T 1-T 2 dual-modal contrast-enhanced magnetic resonance imaging (MRI) and combined with Matrix Laboratory (MATLAB)-based image fusion for staging liver fibrosis in the rat model. Firstly, SPIO@PEG was synthesized and characterized with physical and biological properties as a T 1-T 2 dual-mode MRI contrast agent. Secondly, in the subsequent MR imaging of liver fibrosis in rats in vivo, conventional T 1 and T 2-weighted imaging, and T 1 and T 2 mapping of the liver pre- and post-intravenous administration of SPIO@PEG were systematically collected and analyzed. Thirdly, by creative design, we fused the T 1 and T 2 mapping images by MATLAB and quantitively measured each rat's hepatic fibrosis positive pixel ratio (PPR). SPIO@PEG was proved to have an ultrafine core size (4.01 ± 0.16 nm), satisfactory biosafety and T 1-T 2 dual-mode contrast effects under a 3.0 T MR scanner (r 2/r 1 = 3.51). According to the image fusion results, the SPIO@PEG contrast-enhanced PPR shows significant differences among different stages of liver fibrosis (P < 0.05). The combination of T 1-T 2 dual-modal SPIO@PEG and MATLAB-based image fusion technology could be a promising method for diagnosing and staging liver fibrosis in the rat model. PPR could also be used as a non-invasive biomarker to diagnose and discriminate the stages of liver fibrosis.

Multimodal medical image fusion using convolutional neural network and extreme learning machine

The emergence of multimodal medical imaging technology greatly increases the accuracy of clinical diagnosis and etiological analysis. Nevertheless, each medical imaging modal unavoidably has its own limitations, so the fusion of multimodal medical images may become an effective solution. In this paper, a novel fusion method on the multimodal medical images exploiting convolutional neural network (CNN) and extreme learning machine (ELM) is proposed. As a typical representative in deep learning, CNN has been gaining more and more popularity in the field of image processing. However, CNN often suffers from several drawbacks, such as high computational costs and intensive human interventions. To this end, the model of convolutional extreme learning machine (CELM) is constructed by incorporating ELM into the traditional CNN model. CELM serves as an important tool to extract and capture the features of the source images from a variety of different angles. The final fused image can be obtained by integrating the significant features together. Experimental results indicate that, the proposed method is not only helpful to enhance the accuracy of the lesion detection and localization, but also superior to the current state-of-the-art ones in terms of both subjective visual performance and objective criteria.

A Deep Probabilistic Sensing and Learning Model for Brain Tumor Classification With Fusion-Net and HFCMIK Segmentation

Goal: Implementation of an artificial intelli gence-based medical diagnosis tool for brain tumor classification, which is called the BTFSC-Net. Methods: Medical images are preprocessed using a hybrid probabilistic wiener filter (HPWF) The deep learning convolutional neural network (DLCNN) was utilized to fuse MRI and CT images with robust edge analysis (REA) properties, which are used to identify the slopes and edges of source images. Then, hybrid fuzzy c-means integrated k-means (HFCMIK) clustering is used to segment the disease affected region from the fused image. Further, hybrid features such as texture, colour, and low-level features are extracted from the fused image by using gray-level cooccurrence matrix (GLCM), redundant discrete wavelet transform (RDWT) descriptors. Finally, a deep learning based probabilistic neural network (DLPNN) is used to classify malignant and benign tumors. The BTFSC-Net attained 99.21% of segmentation accuracy and 99.46% of classification accuracy. Conclusions: The simulations showed that BTFSC-Net outperformed as compared to existing methods.

Multimodal Brain Image Fusion Based on Improved Rolling Guidance Filter and Wiener Filter

Medical image fusion technology can integrate complementary information from different modality medical images to provide a more complete and accurate description of the specific diagnosed object, which is very helpful for image-guided clinical diagnosis and treatment. This paper proposes an effective brain image fusion framework based on improved rolling guidance filter (IRGF). Firstly, input images are decomposed into base layers and detail layers using the IRGF and Wiener filter. Secondly, the visual saliency maps of the input image are computed by pixel-level saliency value, and the weight maps of detail layers are constructed by max-absolute strategy and are further smoothed with Gaussian filter, the purpose of which is to make the fused image appear more naturally and more suitable for human visual perception. Lastly, base layers are fused by visual saliency map based fusion rule and the corresponding weight maps from detail layers are fused by the weighted least squares optimization scheme. Experimental results testify that our method is superior to some state-of-the-art methods in both subjective and objective assessments.

Pan-Logical Probabilistic Algorithms Based on Convolutional Neural Networks

A brand-new kind of flexible logic system called universal logic aims to address a variety of uncertain problems. In this study, the role of convolutional neural networks in assessing probabilistic pan-logic algorithms is investigated. A generic logic probability algorithm analysis based on a convolutional neural network is suggested due to the unpredictable outputs of the probabilistic algorithm and the difficulty of its analysis. The stochastic gradient descent technique and the error backpropagation algorithm are used to investigate the broad logic probability algorithm (SGD). The experimental data presented in this research show that the BP algorithm of the convolutional neural network has an accuracy rate of 89 percent when analysing the experimental data. As there are more experimental iterations, the error will go down. The SGD method proves that raising the algorithm's learning rate reduces the loss value of the function. The loss value can be as low as 100%, and the algorithm analysis is closer to the real.

DMC-Fusion: Deep Multi-Cascade Fusion With Classifier-Based Feature Synthesis for Medical Multi-Modal Images

Multi-modal medical image fusion is a challenging yet important task for precision diagnosis and surgical planning in clinical practice. Although single feature fusion strategy such as Densefuse has achieved inspiring performance, it tends to be not fully preserved for the source image features. In this paper, a deep multi-fusion framework with classifier-based feature synthesis is proposed to automatically fuse multi-modal medical images. It consists of a pre-trained autoencoder based on dense connections, a feature classifier and a multi-cascade fusion decoder with separately fusing high-frequency and low-frequency. The encoder and decoder are transferred from MS-COCO datasets and pre-trained simultaneously on multi-modal medical image public datasets to extract features. The feature classification is conducted through Gaussian high-pass filtering and the peak signal to noise ratio thresholding, then feature maps in each layer of the pre-trained Dense-Block and decoder are divided into high-frequency and low-frequency sequences. Specifically, in proposed feature fusion block, parameter-adaptive pulse coupled neural network and l₁-weighted are employed to fuse high-frequency and low-frequency, respectively. Finally, we design a novel multi-cascade fusion decoder on total decoding feature stage to selectively fuse useful information from different modalities. We also validate our approach for the brain disease classification using the fused images, and a statistical significance test is performed to illustrate that the improvement in classification performance is due to the fusion. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance in both qualitative and quantitative evaluations.

HID: The Hybrid Image Decomposition Model for MRI and CT Fusion

Multimodal medical image fusion can combine salient information from different source images of the same part and reduce the redundancy of information. In this paper, an efficient hybrid image decomposition (HID) method is proposed. It combines the advantages of spatial domain and transform domain methods and breaks through the limitations of the algorithms based on single category features. The accurate separation of base layer and texture details is conducive to the better effect of the fusion rules. First, the source anatomical images are decomposed into a series of high frequencies and a low frequency via nonsubsampled shearlet transform (NSST). Second, the low frequency is further decomposed using the designed optimization model based on structural similarity and structure tensor to get an energy texture layer and a base layer. Then, the modified choosing maximum (MCM) is designed to fuse base layers. The sum of modified Laplacian (SML) is used to fuse high frequencies and energy texture layers. Finally, the fused low frequency can be obtained by adding fused energy texture layer and base layer. And the fused image is reconstructed by the inverse NSST. The superiority of the proposed method is verified by amounts of experiments on 50 pairs of magnetic resonance imaging (MRI) images and computed tomography (CT) images and others, and compared with 12 state-of-the-art medical image fusion methods. It is demonstrated that the proposed hybrid decomposition model has a better ability to extract texture information than conventional ones.

OPTIMIZED DUAL-TREE COMPLEX WAVELET TRANSFORM AND FUZZY ENTROPY FOR MULTI-MODAL MEDICAL IMAGE FUSION: A HYBRID META-HEURISTIC CONCEPT

In recent times, multi-modal medical image fusion has emerged as an important medical application tool. An important goal is to fuse the multi-modal medical images from diverse imaging modalities into a single fused image. The physicians broadly utilize this for precise identification and treatment of diseases. This medical image fusion approach will help the physician perform the combined diagnosis, interventional treatment, pre-operative planning, and intra-operative guidance in various medical applications by developing the corresponding information from clinical images through different modalities. In this paper, a novel multi-modal medical image fusion method is adopted using the intelligent method. Initially, the images from two different modalities are applied with optimized Dual-Tree Complex Wavelet Transform (DT-CWT) for splitting the images into high-frequency subbands and low-frequency subbands. As an improvement to the conventional DT-CWT, the filter coefficients are optimized by the hybrid meta-heuristic algorithm named as Hybrid Beetle and Salp Swarm Optimization (HBSSO) by merging the Salp Swarm Algorithm (SSA), and Beetle Swarm Optimization (BSO). Moreover, the fusion of the source images’ high-frequency subbands was done by the optimized type-2 Fuzzy Entropy. The upper and lower membership limits are optimized by the same hybrid HBSSO. The optimized type-2 fuzzy Entropy automatically selects high-frequency coefficients. Also, the fusion of the low-frequency sub-images is performed by the Averaging approach. Further, the inverse optimized DT-CWT on the fused image sets helps to obtain the final fused medical image. The main objective of the optimized DT-CWT and optimized type-2 fuzzy Entropy is to maximize the SSIM. The experimental results confirm that the developed approach outperforms the existing fusion algorithms in diverse performance measures.

A practical PET/CT data visualization method with dual-threshold PET colorization and image fusion

Multi-modal medical imaging has emerged as a general trend in clinical diagnosis and treatment planning. In recent years, great efforts have been made to investigate and develop dual-modality scanners, among which PET/CT is the most widespread one in clinical practice. In this paper, we propose a simple yet effective PET/CT data visualization method that can integrate these two modalities into composite data for better observation. The proposed method consists of three main steps. First, a PET data colorization approach is presented based on a dual-threshold scheme, which applies a pair of high and low thresholds to colorize the PET image. Then, to extract functional information from the PET image more adequately, unlike traditional blending fashion that directly uses the CT image as underlay, we merge the CT and the PET images with a Laplacian pyramid (LP)-based image fusion approach to generate the underlay. Finally, the visualization result is obtained by blending the fused image and the colorized PET image. Experiments are conducted on 5 sets of PET/CT scans that contain 200 paired slices in total. The ClearCanvas software and the method using the presented PET colorization approach but with the CT image as underlay are adopted for comparison. Experimental results demonstrate that the proposed method can achieve more promising performance in terms of both visual perception and quantitative assessment. The code of the proposed method has been made available online athttps://github.com/yuliu316316/Visualization.