Nonlinear Mean Shift over Riemannian Manifolds

Few-shot remote sensing image scene classification based on multiscale covariance metric network (MCMNet)

Few-shot learning (FSL) is a paradigm that simulates the fast learning ability of human beings, which can learn the feature differences between two groups of small-scale samples with common label space, and the label space of the training set and the test set is not repeated. By this way, it can quickly identify the categories of the unseen image in the test set. This method is widely used in image scene recognition, and it is expected to overcome difficulties of scarce annotated samples in remote sensing (RS). However, among most existing FSL methods, images were embed into Euclidean space, and the similarity between features at the last layer of deep network were measured by Euclidean distance. It is difficult to measure the inter-class similarity and intra-class difference of RS images. In this paper, we propose a multi-scale covariance network (MCMNet) for the application of remote sensing scene classification (RSSC). Taking Conv64F as the backbone, we mapped the features of the 1, 2, and 4 layers of the network to the manifold space by constructing a regional covariance matrix to form a covariance network with different scales. For each layer of features, we introduce the center in manifold space as a prototype for different categories of features. We simultaneously measure the similarity of three prototypes on the manifold space with different scales to form three loss functions and optimize the whole network by episodic training strategy. We conducted comparative experiments on three public datasets. The results show that the classification accuracy (CA) of our proposed method is from 1.35 % to 2.36% higher than that of the most excellent method, which demonstrates that the performance of MCMNet outperforms other methods.

Learning Log-Determinant Divergences for Positive Definite Matrices

Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det divergence, which is a meta-divergence parametrized by scalars α and β, subsuming a wide family of popular information divergences on SPD matrices for distinct and discrete values of these parameters. Our key idea is to cast these parameters in a continuum and learn them from data. We systematically extend this idea to learn vector-valued parameters, thereby increasing the expressiveness of the underlying non-linear measure. We conjoin the divergence learning problem with several standard tasks in machine learning, including supervised discriminative dictionary learning and unsupervised SPD matrix clustering. We present Riemannian gradient descent schemes for optimizing our formulations efficiently, and show the usefulness of our method on eight standard computer vision tasks.

Uncovering shape signatures of resting-state functional connectivity by geometric deep learning on Riemannian manifold

Functional neural activities manifest geometric patterns, as evidenced by the evolving network topology of functional connectivities (FC) even in the resting state. In this work, we propose a novel manifold‐based geometric neural network for functional brain networks (called “Geo‐Net4Net” for short) to learn the intrinsic low‐dimensional feature representations of resting‐state brain networks on the Riemannian manifold. This tool allows us to answer the scientific question of how the spontaneous fluctuation of FC supports behavior and cognition. We deploy a set of positive maps and rectified linear unit (ReLU) layers to uncover the intrinsic low‐dimensional feature representations of functional brain networks on the Riemannian manifold taking advantage of the symmetric positive‐definite (SPD) form of the correlation matrices. Due to the lack of well‐defined ground truth in the resting state, existing learning‐based methods are limited to unsupervised methodologies. To go beyond this boundary, we propose to self‐supervise the feature representation learning of resting‐state functional networks by leveraging the task‐based counterparts occurring before and after the underlying resting state. With this extra heuristic, our Geo‐Net4Net allows us to establish a more reasonable understanding of resting‐state FCs by capturing the geometric patterns (aka. spectral/shape signature) associated with resting states on the Riemannian manifold. We have conducted extensive experiments on both simulated data and task‐based functional resonance magnetic imaging (fMRI) data from the Human Connectome Project (HCP) database, where our Geo‐Net4Net not only achieves more accurate change detection results than other state‐of‐the‐art counterpart methods but also yields ubiquitous geometric patterns that manifest putative insights into brain function.

Feature Point Matching Method for Aerial Image Based on Recursive Diffusion Algorithm

Aerial images are large-scale and susceptible to light. Traditional image feature point matching algorithms cannot achieve satisfactory matching accuracy for aerial images. This paper proposes a recursive diffusion algorithm, which is scale-invariant and can be used to extract symmetrical areas of different images. This narrows the matching range of feature points by extracting high-density areas of the image and improving the matching accuracy through correlation analysis of high-density areas. Through experimental comparison, it can be found that the recursive diffusion algorithm has more advantages compared to the correlation coefficient method and the mean shift algorithm when matching accuracy of aerial images, especially when the light of aerial images changes greatly.

Quantized Residual Preference Based Linkage Clustering for Model Selection and Inlier Segmentation in Geometric Multi-Model Fitting

In this paper, quantized residual preference is proposed to represent the hypotheses and the points for model selection and inlier segmentation in multi-structure geometric model fitting. First, a quantized residual preference is proposed to represent the hypotheses. Through a weighted similarity measurement and linkage clustering, similar hypotheses are put into one cluster, and hypotheses with good quality are selected from the clusters as the model selection results. After this, the quantized residual preference is also used to present the data points, and through the linkage clustering, the inliers belonging to the same model can be separated from the outliers. To exclude outliers as many as possible, an iterative sampling and clustering process is performed within the clustering process until the clusters are stable. The experiments undertake indicate that the proposed method performs even better on real data than the some state-of-the-art methods.

Searching for Representative Modes on Hypergraphs for Robust Geometric Model Fitting

In this paper, we propose a simple and effective geometric model fitting method to fit and segment multi-structure data even in the presence of severe outliers. We cast the task of geometric model fitting as a representative mode-seeking problem on hypergraphs. Specifically, a hypergraph is first constructed, where the vertices represent model hypotheses and the hyperedges denote data points. The hypergraph involves higher-order similarities (instead of pairwise similarities used on a simple graph), and it can characterize complex relationships between model hypotheses and data points. In addition, we develop a hypergraph reduction technique to remove "insignificant" vertices while retaining as many "significant" vertices as possible in the hypergraph. Based on the simplified hypergraph, we then propose a novel mode-seeking algorithm to search for representative modes within reasonable time. Finally, the proposed mode-seeking algorithm detects modes according to two key elements, i.e., the weighting scores of vertices and the similarity analysis between vertices. Overall, the proposed fitting method is able to efficiently and effectively estimate the number and the parameters of model instances in the data simultaneously. Experimental results demonstrate that the proposed method achieves significant superiority over several state-of-the-art model fitting methods on both synthetic data and real images.

The Riemannian Potato Field: A Tool for Online Signal Quality Index of EEG

Electroencephalographic (EEG) recordings are contaminated by instrumental, environmental, and biological artifacts, resulting in low signal-to-noise ratio. Artifact detection is a critical task for real-time applications where the signal is used to give a continuous feedback to the user. In these applications, it is therefore necessary to estimate online a signal quality index (SQI) in order to stop the feedback when the signal quality is unacceptable. In this paper, we introduce the Riemannian potato field (RPF) algorithm as such SQI. It is a generalization and extensionof theRiemannian potato, a previouslypublished real-time artifact detection algorithm, whose performance is degraded as the number of channels increases. The RPF overcomes this limitation by combining the outputs of several smaller potatoes into a unique SQI resulting in a higher sensitivity and specificity, regardless of the number of electrodes. We demonstrate these results on a clinical dataset totalizing more than 2200 h of EEG recorded at home, that is, in a non-controlled environment.

A Comprehensive Look at Coding Techniques on Riemannian Manifolds

Core to many learning pipelines is visual recognition such as image and video classification. In such applications, having a compact yet rich and informative representation plays a pivotal role. An underlying assumption in traditional coding schemes [e.g., sparse coding (SC)] is that the data geometrically comply with the Euclidean space. In other words, the data are presented to the algorithm in vector form and Euclidean axioms are fulfilled. This is of course restrictive in machine learning, computer vision, and signal processing, as shown by a large number of recent studies. This paper takes a further step and provides a comprehensive mathematical framework to perform coding in curved and non-Euclidean spaces, i.e., Riemannian manifolds. To this end, we start by the simplest form of coding, namely, bag of words. Then, inspired by the success of vector of locally aggregated descriptors in addressing computer vision problems, we will introduce its Riemannian extensions. Finally, we study Riemannian form of SC, locality-constrained linear coding, and collaborative coding. Through rigorous tests, we demonstrate the superior performance of our Riemannian coding schemes against the state-of-the-art methods on several visual classification tasks, including head pose classification, video-based face recognition, and dynamic scene recognition.

RAPID: Measuring Deformation of Biological Tissues from MR Images Through the Riemannian Pseudo Kernel

Due to the nonlinear deformation of nonrigid and nonuniform tissues, it is challenging to accurately measure the displacements of feature points distributed on the inner parts, boundaries, and separatrices of tissue layers. To address this challenge, we propose a feature point matching technique called RAPID to measure MR 2D slice deformation of nonuniform and nonrigid biological tissues. We propose to use the covariance of several neighboring point statistics computed around a keypoint, as the keypoint descriptor. Inspired by the kernel methods, we advocate adopting a Riemannian pseudo kernel to map SPD matrices to a high dimensional Hilbert space, where the Euclidean geometry applies. We compare our RAPID with two existing schemes (i.e., SIFT and SURF). Our experimental results show that our RAPID is superior to SIFT and SURF, because the benefits offered by RAPID are two-fold. First, our RAPID increases the number of matched data points. Second, RAPID substantially improves the key-point matching accuracy of SIFT and SURF.

Transformations Based on Continuous Piecewise-Affine Velocity Fields

We propose novel finite-dimensional spaces of well-behaved transformations. The latter are obtained by (fast and highly-accurate) integration of continuous piecewise-affine velocity fields. The proposed method is simple yet highly expressive, effortlessly handles optional constraints (e.g., volume preservation and/or boundary conditions), and supports convenient modeling choices such as smoothing priors and coarse-to-fine analysis. Importantly, the proposed approach, partly due to its rapid likelihood evaluations and partly due to its other properties, facilitates tractable inference over rich transformation spaces, including using Markov-Chain Monte-Carlo methods. Its applications include, but are not limited to: monotonic regression (more generally, optimization over monotonic functions); modeling cumulative distribution functions or histograms; time-warping; image warping; image registration; real-time diffeomorphic image editing; data augmentation for image classifiers. Our GPU-based code is publicly available.

Bayesian Nonparametric Clustering for Positive Definite Matrices

Symmetric Positive Definite (SPD) matrices emerge as data descriptors in several applications of computer vision such as object tracking, texture recognition, and diffusion tensor imaging. Clustering these data matrices forms an integral part of these applications, for which soft-clustering algorithms (K-Means, expectation maximization, etc.) are generally used. As is well-known, these algorithms need the number of clusters to be specified, which is difficult when the dataset scales. To address this issue, we resort to the classical nonparametric Bayesian framework by modeling the data as a mixture model using the Dirichlet process (DP) prior. Since these matrices do not conform to the Euclidean geometry, rather belongs to a curved Riemannian manifold,existing DP models cannot be directly applied. Thus, in this paper, we propose a novel DP mixture model framework for SPD matrices. Using the log-determinant divergence as the underlying dissimilarity measure to compare these matrices, and further using the connection between this measure and the Wishart distribution, we derive a novel DPM model based on the Wishart-Inverse-Wishart conjugate pair. We apply this model to several applications in computer vision. Our experiments demonstrate that our model is scalable to the dataset size and at the same time achieves superior accuracy compared to several state-of-the-art parametric and nonparametric clustering algorithms.

Common Visual Pattern Discovery via Nonlinear Mean Shift Clustering

Discovering common visual patterns (CVPs) from two images is a challenging task due to the geometric and photometric deformations as well as noises and clutters. The problem is generally boiled down to recovering correspondences of local invariant features, and the conventionally addressed by graph-based quadratic optimization approaches, which often suffer from high computational cost. In this paper, we propose an efficient approach by viewing the problem from a novel perspective. In particular, we consider each CVP as a common object in two images with a group of coherently deformed local regions. A geometric space with matrix Lie group structure is constructed by stacking up transformations estimated from initially appearance-matched local interest region pairs. This is followed by a mean shift clustering stage to group together those close transformations in the space. Joining regions associated with transformations of the same group together within each input image forms two large regions sharing similar geometric configuration, which naturally leads to a CVP. To account for the non-Euclidean nature of the matrix Lie group, mean shift vectors are derived in the corresponding Lie algebra vector space with a newly provided effective distance measure. Extensive experiments on single and multiple common object discovery tasks as well as near-duplicate image retrieval verify the robustness and efficiency of the proposed approach.

Shape component analysis: structure-preserving dimension reduction on biological shape spaces

MOTIVATION

Quantitative shape analysis is required by a wide range of biological studies across diverse scales, ranging from molecules to cells and organisms. In particular, high-throughput and systems-level studies of biological structures and functions have started to produce large volumes of complex high-dimensional shape data. Analysis and understanding of high-dimensional biological shape data require dimension-reduction techniques.

RESULTS

We have developed a technique for non-linear dimension reduction of 2D and 3D biological shape representations on their Riemannian spaces. A key feature of this technique is that it preserves distances between different shapes in an embedded low-dimensional shape space. We demonstrate an application of this technique by combining it with non-linear mean-shift clustering on the Riemannian spaces for unsupervised clustering of shapes of cellular organelles and proteins.

AVAILABILITY AND IMPLEMENTATION

Source code and data for reproducing results of this article are freely available at https://github.com/ccdlcmu/shape_component_analysis_Matlab The implementation was made in MATLAB and supported on MS Windows, Linux and Mac OS.

CONTACT

geyang@andrew.cmu.edu.

Clustering High-Dimensional Landmark-based Two-dimensional Shape Data‡

An important goal in image analysis is to cluster and recognize objects of interest according to the shapes of their boundaries. Clustering such objects faces at least four major challenges including a curved shape space, a high-dimensional feature space, a complex spatial correlation structure, and shape variation associated with some covariates (e.g., age or gender). The aim of this paper is to develop a penalized model-based clustering framework to cluster landmark-based planar shape data, while explicitly addressing these challenges. Specifically, a mixture of offset-normal shape factor analyzers (MOSFA) is proposed with mixing proportions defined through a regression model (e.g., logistic) and an offset-normal shape distribution in each component for data in the curved shape space. A latent factor analysis model is introduced to explicitly model the complex spatial correlation. A penalized likelihood approach with both adaptive pairwise fusion Lasso penalty function and L₂ penalty function is used to automatically realize variable selection via thresholding and deliver a sparse solution. Our real data analysis has confirmed the excellent finite-sample performance of MOSFA in revealing meaningful clusters in the corpus callosum shape data obtained from the Attention Deficit Hyperactivity Disorder-200 (ADHD-200) study.

A Geometric Particle Filter for Template-Based Visual Tracking

Existing approaches to template-based visual tracking, in which the objective is to continuously estimate the spatial transformation parameters of an object template over video frames, have primarily been based on deterministic optimization, which as is well-known can result in convergence to local optima. To overcome this limitation of the deterministic optimization approach, in this paper we present a novel particle filtering approach to template-based visual tracking. We formulate the problem as a particle filtering problem on matrix Lie groups, specifically the three-dimensional Special Linear group SL(3) and the two-dimensional affine group Aff(2). Computational performance and robustness are enhanced through a number of features: (i) Gaussian importance functions on the groups are iteratively constructed via local linearization; (ii) the inverse formulation of the Jacobian calculation is used; (iii) template resizing is performed; and (iv) parent-child particles are developed and used. Extensive experimental results using challenging video sequences demonstrate the enhanced performance and robustness of our particle filtering-based approach to template-based visual tracking. We also show that our approach outperforms several state-of-the-art template-based visual tracking methods via experiments using the publicly available benchmark data set.

Nonlinear dynamic model for visual object tracking on Grassmann manifolds with partial occlusion handling

This paper proposes a novel Bayesian online learning and tracking scheme for video objects on Grassmann manifolds. Although manifold visual object tracking is promising, large and fast nonplanar (or out-of-plane) pose changes and long-term partial occlusions of deformable objects in video remain a challenge that limits the tracking performance. The proposed method tackles these problems with the main novelties on: 1) online estimation of object appearances on Grassmann manifolds; 2) optimal criterion-based occlusion handling for online updating of object appearances; 3) a nonlinear dynamic model for both the appearance basis matrix and its velocity; and 4) Bayesian formulations, separately for the tracking process and the online learning process, that are realized by employing two particle filters: one is on the manifold for generating appearance particles and another on the linear space for generating affine box particles. Tracking and online updating are performed in an alternating fashion to mitigate the tracking drift. Experiments using the proposed tracker on videos captured by a single dynamic/static camera have shown robust tracking performance, particularly for scenarios when target objects contain significant nonplanar pose changes and long-term partial occlusions. Comparisons with eight existing state-of-the-art/most relevant manifold/nonmanifold trackers with evaluations have provided further support to the proposed scheme.

Simultaneously fitting and segmenting multiple-structure data with outliers

We propose a robust fitting framework, called Adaptive Kernel-Scale Weighted Hypotheses (AKSWH), to segment multiple-structure data even in the presence of a large number of outliers. Our framework contains a novel scale estimator called Iterative Kth Ordered Scale Estimator (IKOSE). IKOSE can accurately estimate the scale of inliers for heavily corrupted multiple-structure data and is of interest by itself since it can be used in other robust estimators. In addition to IKOSE, our framework includes several original elements based on the weighting, clustering, and fusing of hypotheses. AKSWH can provide accurate estimates of the number of model instances and the parameters and the scale of each model instance simultaneously. We demonstrate good performance in practical applications such as line fitting, circle fitting, range image segmentation, homography estimation, and two--view-based motion segmentation, using both synthetic data and real images.

Statistical computations on Grassmann and Stiefel manifolds for image and video-based recognition

In this paper, we examine image and video-based recognition applications where the underlying models have a special structure—the linear subspace structure. We discuss how commonly used parametric models for videos and image sets can be described using the unified framework of Grassmann and Stiefel manifolds. We first show that the parameters of linear dynamic models are finite-dimensional linear subspaces of appropriate dimensions. Unordered image sets as samples from a finite-dimensional linear subspace naturally fall under this framework. We show that an inference over subspaces can be naturally cast as an inference problem on the Grassmann manifold. To perform recognition using subspace-based models, we need tools from the Riemannian geometry of the Grassmann manifold. This involves a study of the geometric properties of the space, appropriate definitions of Riemannian metrics, and definition of geodesics. Further, we derive statistical modeling of inter and intraclass variations that respect the geometry of the space. We apply techniques such as intrinsic and extrinsic statistics to enable maximum-likelihood classification. We also provide algorithms for unsupervised clustering derived from the geometry of the manifold. Finally, we demonstrate the improved performance of these methods in a wide variety of vision applications such as activity recognition, video-based face recognition, object recognition from image sets, and activity-based video clustering.