Adaptive Discrete Hypergraph Matching

Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem With Extension to Hypergraph and Multiple-Graph Matching

Graph matching involves combinatorial optimization based on edge-to-edge affinity matrix, which can be generally formulated as Lawler's quadratic assignment problem (QAP). This paper presents a QAP network directly learning with the affinity matrix (equivalently the association graph) whereby the matching problem is translated into a constrained vertex classification task. The association graph is learned by an embedding network for vertex classification, followed by Sinkhorn normalization and a cross-entropy loss for end-to-end learning. We further improve the embedding model on association graph by introducing Sinkhorn based matching-aware constraint, as well as dummy nodes to deal with unequal sizes of graphs. To our best knowledge, this is one of the first network to directly learn with the general Lawler's QAP. In contrast, recent deep matching methods focus on the learning of node/edge features in two graphs respectively. We also show how to extend our network to hypergraph matching, and matching of multiple graphs. Experimental results on both synthetic graphs and real-world images show its effectiveness. For pure QAP tasks on synthetic data and QAPLIB benchmark, our method can perform competitively and even surpass state-of-the-art graph matching and QAP solvers with notable less time cost. We provide a project homepage at http://thinklab.sjtu.edu.cn/project/NGM/index.html.

Small Object Detection via Pixel Level Balancing With Applications to Blood Cell Detection

Object detection technology has been widely used in medical field, such as detecting the images of blood cell to count the changes and distribution for assisting the diagnosis of diseases. However, detecting small objects is one of the most challenging and important problems especially in medical scenarios. Most of the objects in medical images are very small but influential. Improving the detection performance of small objects is a very meaningful topic for medical detection. Current researches mainly focus on the extraction of small object features and data augmentation for small object samples, all of these researches focus on extracting the feature space of small objects better. However, in the training process of a detection model, objects of different sizes are mixed together, which may interfere with each other and affect the performance of small object detection. In this paper, we propose a method called pixel level balancing (PLB), which takes into account the number of pixels contained in the detection box as an impact factor to characterize the size of the inspected objects, and uses this as an impact factor. The training loss of each object of different size is adjusted by a weight dynamically, so as to improve the accuracy of small object detection. Finally, through experiments, we demonstrate that the size of objects in object detection interfere with each other. So that we can improve the accuracy of small object detection through PLB operation. This method can perform well with blood cell detection in our experiments.

Elastic Net Constraint-Based Tensor Model for High-Order Graph Matching

The procedure of establishing the correspondence between two sets of feature points is important in computer vision applications. In this article, an elastic net constraint-based tensor model is proposed for high-order graph matching. To control the tradeoff between the sparsity and the accuracy of the matching results, an elastic net constraint is introduced into the tensor-based graph matching model. Then, a nonmonotone spectral projected gradient (NSPG) method is derived to solve the proposed matching model. During the optimization of using NSPG, we propose an algorithm to calculate the projection on the feasible convex sets of elastic net constraint. Further, the global convergence of solving the proposed model using the NSPG method was proved. The superiority of the proposed method is verified through experiments on the synthetic data and natural images.

An Accurate and Efficient Voting Scheme for a Maximally All-Inlier 3D Correspondence Set

We present a highly accurate and efficient, yet simple, two-stage voting scheme for distinguishing inlier 3D correspondences by densely assessing and ranking their local and global geometric consistencies. The strength of the proposed method stems from both the novel idea of post-validated voting set, as well as single-point superimposition transforms, which are computationally cheap and avoid rotational ambiguities. Using a well-known dataset consisting of various 3D models and numerous scenes that include different occlusion rates, the proposed scheme is evaluated against state-of-the-art 3D voting schemes, in terms of both the correspondence PR (precision-recall) AUC (area under curve), and the execution time. A total of 374 experiments were conducted for each method, which involved a combination of four models, 50 scenes, and two down-samplings. The proposed scheme outperforms the state-of-the-art 3D voting schemes in terms of both accuracy and speed. Quantitatively, the proposed scheme scores 97.0% ±12.9% on the PR AUC metric, averaged over all of the experiments, while the two state-of-the-art schemes score 74.2% ±22.2% and 78.3% ±26.4%. Furthermore, the proposed scheme requires only 24.1% ±6.0% of the time consumed by the fastest state-of-the-art scheme. The proposed voting scheme also demonstrates high robustness against occlusions and scarce inliers.

Spatial Location in Integrated Circuits through Infrared Microscopy

In this paper, we present an infrared microscopy based approach for structures' location in integrated circuits, to automate their secure characterization. The use of an infrared sensor is the key device for internal integrated circuit inspection. Two main issues are addressed. The first concerns the scan of integrated circuits using a motorized optical system composed of an infrared uncooled camera combined with an optical microscope. An automated system is required to focus the conductive tracks under the silicon layer. It is solved by an autofocus system analyzing the infrared images through a discrete polynomial image transform which allows an accurate features detection to build a focus metric robust against specific image degradation inherent to the acquisition context. The second issue concerns the location of structures to be characterized on the conductive tracks. Dealing with a large amount of redundancy and noise, a graph-matching method is presented-discriminating graph labels are developed to overcome the redundancy, while a flexible assignment optimizer solves the inexact matching arising from noises on graphs. The resulting automated location system brings reproducibility for secure characterization of integrated systems, besides accuracy and time speed increase.

Group Reidentification with Multigrained Matching and Integration

The task of reidentifying groups of people under different camera views is an important yet less-studied problem. Group reidentification (Re-ID) is a very challenging task since it is not only adversely affected by common issues in traditional single-object Re-ID problems, such as viewpoint and human pose variations, but also suffers from changes in group layout and group membership. In this paper, we propose a novel concept of group granularity by characterizing a group image by multigrained objects: individual people and subgroups of two and three people within a group. To achieve robust group Re-ID, we first introduce multigrained representations which can be extracted via the development of two separate schemes, that is, one with handcrafted descriptors and another with deep neural networks. The proposed representation seeks to characterize both appearance and spatial relations of multigrained objects, and is further equipped with importance weights which capture variations in intragroup dynamics. Optimal group-wise matching is facilitated by a multiorder matching process which, in turn, dynamically updates the importance weights in iterative fashion. We evaluated three multicamera group datasets containing complex scenarios and large dynamics, with experimental results demonstrating the effectiveness of our approach.

Joint Transformation Learning via the L2,1-Norm Metric for Robust Graph Matching

Establishing correspondence between two given geometrical graph structures is an important problem in computer vision and pattern recognition. In this paper, we propose a robust graph matching (RGM) model to improve the effectiveness and robustness on the matching graphs with deformations, rotations, outliers, and noise. First, we embed the joint geometric transformation into the graph matching model, which performs unary matching over graph nodes and local structure matching over graph edges simultaneously. Then, the L_2,1 -norm is used as the similarity metric in the presented RGM to enhance the robustness. Finally, we derive an objective function which can be solved by an effective optimization algorithm, and theoretically prove the convergence of the proposed algorithm. Extensive experiments on various graph matching tasks, such as outliers, rotations, and deformations show that the proposed RGM model achieves competitive performance compared to the existing methods.

A new method to predict anomaly in brain network based on graph deep learning

Functional magnetic resonance imaging a neuroimaging technique which is used in brain disorders and dysfunction studies, has been improved in recent years by mapping the topology of the brain connections, named connectopic mapping. Based on the fact that healthy and unhealthy brain regions and functions differ slightly, studying the complex topology of the functional and structural networks in the human brain is too complicated considering the growth of evaluation measures. One of the applications of irregular graph deep learning is to analyze the human cognitive functions related to the gene expression and related distributed spatial patterns. Since a variety of brain solutions can be dynamically held in the neuronal networks of the brain with different activity patterns and functional connectivity, both node-centric and graph-centric tasks are involved in this application. In this study, we used an individual generative model and high order graph analysis for the region of interest recognition areas of the brain with abnormal connection during performing certain tasks and resting-state or decompose irregular observations. Accordingly, a high order framework of Variational Graph Autoencoder with a Gaussian distributer was proposed in the paper to analyze the functional data in brain imaging studies in which Generative Adversarial Network is employed for optimizing the latent space in the process of learning strong non-rigid graphs among large scale data. Furthermore, the possible modes of correlations were distinguished in abnormal brain connections. Our goal was to find the degree of correlation between the affected regions and their simultaneous occurrence over time. We can take advantage of this to diagnose brain diseases or show the ability of the nervous system to modify brain topology at all angles and brain plasticity according to input stimuli. In this study, we particularly focused on Alzheimer's disease.

Texture Synthesis Based Thyroid Nodule Detection From Medical Ultrasound Images: Interpreting and Suppressing the Adversarial Effect of In-place Manual Annotation

Deep learning method have been offering promising solutions for medical image processing, but failing to understand what features in the input image are captured and whether certain artifacts are mistakenly included in the model, thus create crucial problems in generalizability of the model. We targeted a common issue of this kind caused by manual annotations appeared in medical image. These annotations are usually made by the doctors at the spot of medical interest and have adversarial effect on many computer vision AI tasks. We developed an inpainting algorithm to remove the annotations and recover the original images. Besides we applied variational information bottleneck method in order to filter out the unwanted features and enhance the robustness of the model. Our impaiting algorithm is extensively tested in object detection in thyroid ultrasound image data. The mAP (mean average precision, with IoU = 0.3) is 27% without the annotation removal. The mAP is 83% if manually removed the annotations using Photoshop and is enhanced to 90% using our inpainting algorithm. Our work can be utilized in future development and evaluation of artificial intelligence models based on medical images with defects.

Robust Feature Matching Using Spatial Clustering with Heavy Outliers

This paper focuses on removing mismatches from given putative feature matches created typically based on descriptor similarity. To achieve this goal, existing attempts usually involve estimating the image transformation under a geometrical constraint, where a pre-defined transformation model is demanded. This severely limits the applicability, as the transformation could vary with different data and is complex and hard to model in many real-world tasks. From a novel perspective, this paper casts the feature matching into a spatial clustering problem with outliers. The main idea is to adaptively cluster the putative matches into several motion consistent clusters together with an outlier/mismatch cluster. To implement the spatial clustering, we customize the classic density based spatial clustering method of applications with noise (DBSCAN) in the context of feature matching, which enables our approach to achieve quasi-linear time complexity. We also design an iterative clustering strategy to promote the matching performance in case of severely degraded data. Extensive experiments on several datasets involving different types of image transformations demonstrate the superiority of our approach over state-of-the-art alternatives. Our approach is also applied to near-duplicate image retrieval and co-segmentation and achieves promising performance.

Spectral-Spatial Attention Networks for Hyperspectral Image Classification

Many deep learning models, such as convolutional neural network (CNN) and recurrent neural network (RNN), have been successfully applied to extracting deep features for hyperspectral tasks. Hyperspectral image classification allows distinguishing the characterization of land covers by utilizing their abundant information. Motivated by the attention mechanism of the human visual system, in this study, we propose a spectral-spatial attention network for hyperspectral image classification. In our method, RNN with attention can learn inner spectral correlations within a continuous spectrum, while CNN with attention is designed to focus on saliency features and spatial relevance between neighboring pixels in the spatial dimension. Experimental results demonstrate that our method can fully utilize the spectral and spatial information to obtain competitive performance.

A Robust Method for Automatic Panoramic UAV Image Mosaic

This paper introduces a robust method for panoramic unmanned aerial vehicle (UAV) image mosaic. In the traditional automatic panoramic image stitching method (Autostitch), it assumes that the camera rotates about its optical centre and the group of transformations the source images may undergo is a special group of homographies. It is rare to get such ideal data in reality. In particular, remote sensing images obtained by UAV do not satisfy such an ideal situation, where the images may not be on a plane yet and even may suffer from nonrigid changes, leading to poor mosaic results. To overcome the above mentioned challenges, in this paper a nonrigid matching algorithm is introduced to the mosaic system to generate accurate feature matching on remote sensing images. We also propose a new strategy for bundle adjustment to make the mosaic system suitable for the UAV image panoramic mosaic effect. Experimental results show that our method outperforms the traditional method and some of the latest methods in terms of visual effect.

Fusing Infrared and Visible Images of Different Resolutions via Total Variation Model

In infrared and visible image fusion, existing methods typically have a prerequisite that the source images share the same resolution. However, due to limitations of hardware devices and application environments, infrared images constantly suffer from markedly lower resolution compared with the corresponding visible images. In this case, current fusion methods inevitably cause texture information loss in visible images or blur thermal radiation information in infrared images. Moreover, the principle of existing fusion rules typically focuses on preserving texture details in source images, which may be inappropriate for fusing infrared thermal radiation information because it is characterized by pixel intensities, possibly neglecting the prominence of targets in fused images. Faced with such difficulties and challenges, we propose a novel method to fuse infrared and visible images of different resolutions and generate high-resolution resulting images to obtain clear and accurate fused images. Specifically, the fusion problem is formulated as a total variation (TV) minimization problem. The data fidelity term constrains the pixel intensity similarity of the downsampled fused image with respect to the infrared image, and the regularization term compels the gradient similarity of the fused image with respect to the visible image. The fast iterative shrinkage-thresholding algorithm (FISTA) framework is applied to improve the convergence rate. Our resulting fused images are similar to super-resolved infrared images, which are sharpened by the texture information from visible images. Advantages and innovations of our method are demonstrated by the qualitative and quantitative comparisons with six state-of-the-art methods on publicly available datasets.

Hyperspectral Unmixing with Bandwise Generalized Bilinear Model

Generalized bilinear model (GBM) has received extensive attention in the field of hyperspectral nonlinear unmixing. Traditional GBM unmixing methods are usually assumed to be degraded only by additive white Gaussian noise (AWGN), and the intensity of AWGN in each band of hyperspectral image (HSI) is assumed to be the same. However, the real HSIs are usually degraded by mixture of various kinds of noise, which include Gaussian noise, impulse noise, dead pixels or lines, stripes, and so on. Besides, the intensity of AWGN is usually different for each band of HSI. To address the above mentioned issues, we propose a novel nonlinear unmixing method based on the bandwise generalized bilinear model (NU-BGBM), which can be adapted to the presence of complex mixed noise in real HSI. Besides, the alternative direction method of multipliers (ADMM) is adopted to solve the proposed NU-BGBM. Finally, extensive experiments are conducted to demonstrate the effectiveness of the proposed NU-BGBM compared with some other state-of-the-art unmixing methods.