Generative Dual-Adversarial Network With Spectral Fidelity and Spatial Enhancement for Hyperspectral Pansharpening

Self-Supervised Multiscale Contrastive and Attention-Guided Gradient Projection Network for Pansharpening

Pansharpening techniques are crucial in remote sensing image processing, with deep learning emerging as the mainstream solution. In this paper, the pansharpening problem is formulated as two optimization subproblems with a solution proposed based on multiscale contrastive learning combined with attention-guided gradient projection networks. First, an efficient and generalized Spectral-Spatial Universal Module (SSUM) is designed and applied to spectral and spatial enhancement modules (SpeEB and SpaEB). Then, the multiscale high-frequency features of PAN and MS images are extracted using discrete wavelet transform (DWT). These features are combined with contrastive learning and residual connection to progressively balance spectral and spatial information. Finally, high-resolution multispectral images are generated through multiple iterations. Experimental results verify that the proposed method outperforms existing approaches in both visual quality and quantitative evaluation metrics.

Cycle-Refined Multidecision Joint Alignment Network for Unsupervised Domain Adaptive Hyperspectral Change Detection

Hyperspectral change detection, which provides abundant information on land cover changes in the Earth's surface, has become one of the most crucial tasks in remote sensing. Recently, deep-learning-based change detection methods have shown remarkable performance, but the acquirement of labeled data is extremely expensive and time-consuming. It is intuitive to learn changes from the scene with sufficient labeled data and adapting them into an unlabeled new scene. However, the nonnegligible domain shift between different scenes leads to inevitable performance degradation. In this article, a cycle-refined multidecision joint alignment network (CMJAN) is proposed for unsupervised domain adaptive hyperspectral change detection, which realizes progressive alignment of the data distributions between the source and target domains with cycle-refined high-confidence labeled samples. There are two key characteristics: 1) progressively mitigate the distribution discrepancy to learn domain-invariant difference feature representation and 2) update the high-confidence training samples of the target domain in a cycle manner. The benefit is that the domain shift between the source and target domains is progressively alleviated to promote change detection performance on the target domain in an unsupervised manner. Experimental results on different datasets demonstrate that the proposed method can achieve better performance than the state-of-the-art change detection methods.

Feature Pyramid Fusion Network for Hyperspectral Pansharpening

Hyperspectral (HS) pansharpening aims at fusing an observed HS image with a panchromatic (PAN) image, to produce an image with the high spectral resolution of the former and the high spatial resolution of the latter. Most of the existing convolutional neural networks (CNNs)-based pansharpening methods reconstruct the desired high-resolution image from the encoded low-resolution (LR) representation. However, the encoded LR representation captures semantic information of the image and is inadequate in reconstructing fine details. How to effectively extract high-resolution and LR representations for high-resolution image reconstruction is the main objective of this article. In this article, we propose a feature pyramid fusion network (FPFNet) for pansharpening, which permits the network to extract multiresolution representations from PAN and HS images in two branches. The PAN branch starts from the high-resolution stream that maintains the spatial resolution of the PAN image and gradually adds LR streams in parallel. The structure of the HS branch remains highly consistent with that of the PAN branch, but starts with the LR stream and gradually adds high-resolution streams. The representations with corresponding resolutions of PAN and HS branches are fused and gradually upsampled in a coarse to fine manner to reconstruct the high-resolution HS image. Experimental results on three datasets demonstrate the significant superiority of the proposed FPFNet over the state-of-the-art methods in terms of both qualitative and quantitative comparisons.

A Multitask Network for Joint Multispectral Pansharpening on Diverse Satellite Data

Despite the rapid advance in multispectral (MS) pansharpening, existing convolutional neural network (CNN)-based methods require training on separate CNNs for different satellite datasets. However, such a single-task learning (STL) paradigm often leads to overlooking any underlying correlations between datasets. Aiming at this challenging problem, a multitask network (MTNet) is presented to accomplish joint MS pansharpening in a unified framework for images acquired by different satellites. Particularly, the pansharpening process of each satellite is treated as a specific task, while MTNet simultaneously learns from all data obtained from these satellites following the multitask learning (MTL) paradigm. MTNet shares the generic knowledge between datasets via task-agnostic subnetwork (TASNet), utilizing task-specific subnetworks (TSSNets) to facilitate the adaptation of such knowledge to a certain satellite. To tackle the limitation of the local connectivity property of the CNN, TASNet incorporates Transformer modules to derive global information. In addition, band-aware dynamic convolutions (BDConvs) are proposed that can accommodate various ground scenes and bands by adjusting their respective receptive field (RF) size. Systematic experimental results over different datasets demonstrate that the proposed approach outperforms the existing state-of-the-art (SOTA) techniques.

Interpretable Model-Driven Deep Network for Hyperspectral, Multispectral, and Panchromatic Image Fusion

Simultaneously fusing hyperspectral (HS), multispectral (MS), and panchromatic (PAN) images brings a new paradigm to generate a high-resolution HS (HRHS) image. In this study, we propose an interpretable model-driven deep network for HS, MS, and PAN image fusion, called HMPNet. We first propose a new fusion model that utilizes a deep before describing the complicated relationship between the HRHS and PAN images owing to their large resolution difference. Consequently, the difficulty of traditional model-based approaches in designing suitable hand-crafted priors can be alleviated because this deep prior is learned from data. We further solve the optimization problem of this fusion model based on the proximal gradient descent (PGD) algorithm, achieved by a series of iterative steps. By unrolling these iterative steps into several network modules, we finally obtain the HMPNet. Therefore, all parameters besides the deep prior are learned in the deep network, simplifying the selection of optimal parameters in the fusion and achieving a favorable equilibrium between the spatial and spectral qualities. Meanwhile, all modules contained in the HMPNet have explainable physical meanings, which can improve its generalization capability. In the experiment, we exhibit the advantages of the HMPNet over other state-of-the-art methods from the aspects of visual comparison and quantitative analysis, where a series of simulated as well as real datasets are utilized for validation.

Heat Transfer-Inspired Network for Image Super-Resolution Reconstruction

Image super-resolution (SR) is a critical image preprocessing task for many applications. How to recover features as accurately as possible is the focus of SR algorithms. Most existing SR methods tend to guide the image reconstruction process with gradient maps, frequency perception modules, etc. and improve the quality of recovered images from the perspective of enhancing edges, but rarely optimize the neural network structure from the system level. In this article, we conduct an in- depth exploration for the inner nature of the SR network structure. In light of the consistency between thermal particles in the thermal field and pixels in the image domain, we propose a novel heat-transfer-inspired network (HTI-Net) for image SR reconstruction based on the theoretical basis of heat transfer. With the finite difference theory, we use a second-order mixed-difference equation to redesign the residual network (ResNet), which can fully integrate multiple information to achieve better feature reuse. In addition, according to the thermal conduction differential equation (TCDE) in the thermal field, the pixel value flow equation (PVFE) in the image domain is derived to mine deep potential feature information. The experimental results on multiple standard databases demonstrate that the proposed HTI-Net has superior edge detail reconstruction effect and parameter performance compared with the existing SR methods. The experimental results on the microscope chip image (MCI) database consisting of realistic low-resolution (LR) and high-resolution (HR) images show that the proposed HTI-Net for image SR reconstruction can improve the effectiveness of the hardware Trojan detection system.

A Spatio-Spectral Fusion Method for Hyperspectral Images Using Residual Hyper-Dense Network

Spatio-spectral fusion of panchromatic (PAN) and hyperspectral (HS) images is of great importance in improving spatial resolution of images acquired by many commercial HS sensors. DenseNets have recently achieved great success for image super-resolution because they facilitate gradient flow by concatenating all the feature outputs in a feedforward manner. In this article, we propose a residual hyper-dense network (RHDN) that extends the DenseNet to solve the spatio-spectral fusion problem. The overall structure of the proposed RHDN method is a two-branch network, which allows the network to capture the features of HS images within and outside the visible range separately. At each branch of the network, a two-stream strategy of feature extraction is designed to process PAN and HS images individually. A convolutional neural network (CNN) with cascade residual hyper-dense blocks (RHDBs), which allows direct connections between the pairs of layers within the same stream and those across different streams, is proposed to learn more complex combinations between the HS and PAN images. The residual learning is adopted to make the network efficient. Extensive benchmark evaluations well demonstrate that the proposed RHDN fusion method yields significant improvements over many widely accepted state-of-the-art approaches.

Full-Resolution Quality Assessment for Pansharpening

A reliable quality assessment procedure for pansharpening methods is of critical importance for the development of the related solutions. Unfortunately, the lack of ground truths to be used as guidance for an objective evaluation has pushed the community to resort to two approaches, which can also be jointly applied. Hence, two kinds of indexes can be found in the literature: (i) reference-based reduced-resolution indexes aimed to assess the synthesis ability; (ii) no-reference subjective quality indexes for full-resolution datasets aimed to assess spectral and spatial consistency. Both reference-based and no-reference indexes present critical shortcomings, which motivate the community to explore new solutions. In this work, we propose an alternative no-reference full-resolution assessment framework. On one side, we introduce a protocol, namely the reprojection protocol, to take care of the spectral consistency issue. On the other side, a new index of the spatial consistency between the pansharpened image and the panchromatic band at full resolution is also proposed. Experimental results carried out on different datasets/sensors demonstrate the effectiveness of the proposed approach.