A New Deep Learning Based Multi-Spectral Image Fusion Method

Improving LIBS-based mineral identification with Raman imaging and spectral knowledge distillation

Combining data from different sensing modalities has been a promising research topic for building better and more reliable data-driven models. In particular, it is known that multimodal spectral imaging can improve the analytical capabilities of standalone spectroscopy techniques through fusion, hyphenation, or knowledge distillation techniques. In this manuscript, we focus on the latter, exploring how one can increase the performance of a Laser-induced Breakdown Spectroscopy system for mineral classification problems using additional spectral imaging techniques. Specifically, focusing on a scenario where Raman spectroscopy delivers accurate mineral classification performance, we show how to deploy a knowledge distillation pipeline where Raman spectroscopy may act as an autonomous supervisor for LIBS. For a case study concerning a challenging Li-bearing mineral identification of spodumene and petalite, our results demonstrate the advantages of this method in improving the performance of a single-technique system. LIBS trained with labels obtained by Raman presents an enhanced classification performance. Furthermore, leveraging the interpretability of the model deployed, the workflow opens opportunities for the deployment of assisted feature discovery pipelines, which may impact future academic and industrial applications.

From sensor fusion to knowledge distillation in collaborative LIBS and hyperspectral imaging for mineral identification

Multimodal spectral imaging offers a unique approach to the enhancement of the analytical capabilities of standalone spectroscopy techniques by combining information gathered from distinct sources. In this manuscript, we explore such opportunities by focusing on two well-known spectral imaging techniques, namely laser-induced breakdown spectroscopy, and hyperspectral imaging, and explore the opportunities of collaborative sensing for a case study involving mineral identification. In specific, the work builds upon two distinct approaches: a traditional sensor fusion, where we strive to increase the information gathered by including information from the two modalities; and a knowledge distillation approach, where the Laser Induced Breakdown spectroscopy is used as an autonomous supervisor for hyperspectral imaging. Our results show the potential of both approaches in enhancing the performance over a single modality sensing system, highlighting, in particular, the advantages of the knowledge distillation framework in maximizing the potential benefits of using multiple techniques to build more interpretable models and paving for industrial applications.

MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

PURPOSE

This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time.

METHODS

We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images.

RESULTS

The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance.

CONCLUSIONS

The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.

Flexible and broadband colloidal quantum dots photodiode array for pixel-level X-ray to near-infrared image fusion

Combining information from multispectral images into a fused image is informative and beneficial for human or machine perception. Currently, multiple photodetectors with different response bands are used, which require complicated algorithms and systems to solve the pixel and position mismatch problem. An ideal solution would be pixel-level multispectral image fusion, which involves multispectral image using the same photodetector and circumventing the mismatch problem. Here we presented the potential of pixel-level multispectral image fusion utilizing colloidal quantum dots photodiode array, with a broadband response range from X-ray to near infrared and excellent tolerance for bending and X-ray irradiation. The colloidal quantum dots photodiode array showed a specific detectivity exceeding 1012 Jones in visible and near infrared range and a favorable volume sensitivity of approximately 2 × 105 μC Gy-1 cm-3 for X-ray irradiation. To showcase the advantages of pixel-level multispectral image fusion, we imaged a capsule enfolding an iron wire and soft plastic, successfully revealing internal information through an X-ray to near infrared fused image.

Gaussian of Differences: A Simple and Efficient General Image Fusion Method

The separate analysis of images obtained from a single source using different camera settings or spectral bands, whether from one or more than one sensor, is quite difficult. To solve this problem, a single image containing all of the distinctive pieces of information in each source image is generally created by combining the images, a process called image fusion. In this paper, a simple and efficient, pixel-based image fusion method is proposed that relies on weighting the edge information associated with each pixel of all of the source images proportional to the distance from their neighbors by employing a Gaussian filter. The proposed method, Gaussian of differences (GD), was evaluated using multi-modal medical images, multi-sensor visible and infrared images, multi-focus images, and multi-exposure images, and was compared to existing state-of-the-art fusion methods by utilizing objective fusion quality metrics. The parameters of the GD method are further enhanced by employing the pattern search (PS) algorithm, resulting in an adaptive optimization strategy. Extensive experiments illustrated that the proposed GD fusion method ranked better on average than others in terms of objective quality metrics and CPU time consumption.

Infrared and Visible Image Fusion for Highlighting Salient Targets in the Night Scene

The goal of infrared and visible image fusion in the night scene is to generate a fused image containing salient targets and rich textural details. However, the existing image fusion methods fail to take the unevenness of nighttime luminance into account. To address the above issue, an infrared and visible image fusion method for highlighting salient targets in the night scene is proposed. First of all, a global attention module is designed, which rescales the weights of different channels after capturing global contextual information. Second, the loss function is divided into the foreground loss and the background loss, forcing the fused image to retain rich texture details while highlighting the salient targets. Finally, a luminance estimation function is introduced to obtain the trade-off control parameters of the foreground loss function based on the nighttime luminance. It can effectively highlight salient targets by retaining the foreground information from the source images. Compared with other advanced methods, the experimental results adequately demonstrate the excellent fusion performance and generalization of the proposed method.

Texture and Materials Image Classification Based on Wavelet Pooling Layer in CNN

Convolutional Neural Networks (CNNs) have recently been proposed as a solution in texture and material classification in computer vision. However, inside CNNs, the internal layers of pooling often cause a loss of information and, therefore, is detrimental to learning the architecture. Moreover, when considering images with repetitive and essential patterns, the loss of this information affects the performance of subsequent stages, such as feature extraction and analysis. In this paper, to solve this problem, we propose a classification system with a new pooling method called Discrete Wavelet Transform Pooling (DWTP). This method is based on the image decomposition into sub-bands, in which the first level sub-band is considered as its output. The objective is to obtain approximation and detail information. As a result, this information can be concatenated in different combinations. In addition, wavelet pooling uses wavelets to reduce the size of the feature map. Combining these methods provides acceptable classification performance for three databases (CIFAR-10, DTD, and FMD). We argue that this helps eliminate overfitting and that the learning graphs reflect that the datasets show learning generalization. Therefore, our results indicate that our method based on wavelet analysis is feasible for texture and material classification. Moreover, in some cases, it outperforms traditional methods.

Deep learning approach for fusion of magnetic resonance imaging-positron emission tomography image based on extract image features using pretrained network (VGG19)

Background:

The fusion of images is an interesting way to display the information of some different images in one image together. In this paper, we present a deep learning network approach for fusion of magnetic resonance imaging (MRI) and positron emission tomography (PET) images.

Methods:

We fused two MRI and PET images automatically with a pretrained convolutional neural network (CNN, VGG19). First, the PET image was converted from red-green-blue space to hue-saturation-intensity space to save the hue and saturation information. We started with extracting features from images by using a pretrained CNN. Then, we used the weights extracted from two MRI and PET images to construct a fused image. Fused image was constructed with multiplied weights to images. For solving the problem of reduced contrast, we added the constant coefficient of the original image to the final result. Finally, quantitative criteria (entropy, mutual information, discrepancy, and overall performance [OP]) were applied to evaluate the results of fusion. We compared the results of our method with the most widely used methods in the spatial and transform domain.

Results:

The quantitative measurement values we used were entropy, mutual information, discrepancy, and OP that were 3.0319, 2.3993, 3.8187, and 0.9899, respectively. The final results showed that our method based on quantitative assessments was the best and easiest way to fused images, especially in the spatial domain.

Conclusion:

It concluded that our method used for MRI-PET image fusion was more accurate.

A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods.