Dense and Sparse Reconstruction Error Based Saliency Descriptor

Part-Object Relational Visual Saliency

Recent years have witnessed a big leap in automatic visual saliency detection attributed to advances in deep learning, especially Convolutional Neural Networks (CNNs). However, inferring the saliency of each image part separately, as was adopted by most CNNs methods, inevitably leads to an incomplete segmentation of the salient object. In this paper, we describe how to use the property of part-object relations endowed by the Capsule Network (CapsNet) to solve the problems that fundamentally hinge on relational inference for visual saliency detection. Concretely, we put in place a two-stream strategy, termed Two-Stream Part-Object RelaTional Network (TSPORTNet), to implement CapsNet, aiming to reduce both the network complexity and the possible redundancy during capsule routing. Additionally, taking into account the correlations of capsule types from the preceding training images, a correlation-aware capsule routing algorithm is developed for more accurate capsule assignments at the training stage, which also speeds up the training dramatically. By exploring part-object relationships, TSPORTNet produces a capsule wholeness map, which in turn aids multi-level features in generating the final saliency map. Experimental results on five widely-used benchmarks show that our framework consistently achieves state-of-the-art performance. The code can be found on https://github.com/liuyi1989/TSPORTNet.

Selecting Post-Processing Schemes for Accurate Detection of Small Objects in Low-Resolution Wide-Area Aerial Imagery

In low-resolution wide-area aerial imagery, object detection algorithms are categorized as feature extraction and machine learning approaches, where the former often requires a post-processing scheme to reduce false detections and the latter demands multi-stage learning followed by post-processing. In this paper, we present an approach on how to select post-processing schemes for aerial object detection. We evaluated combinations of each of ten vehicle detection algorithms with any of seven post-processing schemes, where the best three schemes for each algorithm were determined using average F-score metric. The performance improvement is quantified using basic information retrieval metrics as well as the classification of events, activities and relationships (CLEAR) metrics. We also implemented a two-stage learning algorithm using a hundred-layer densely connected convolutional neural network for small object detection and evaluated its degree of improvement when combined with the various post-processing schemes. The highest average F-scores after post-processing are 0.902, 0.704 and 0.891 for the Tucson, Phoenix and online VEDAI datasets, respectively. The combined results prove that our enhanced three-stage post-processing scheme achieves a mean average precision (mAP) of 63.9% for feature extraction methods and 82.8% for the machine learning approach.

Hierarchical Edge Refinement Network for Saliency Detection

At present, most saliency detection methods are based on fully convolutional neural networks (FCNs). However, FCNs usually blur the edges of salient objects. Due to that, the multiple convolution and pooling operations of the FCNs will limit the spatial resolution of the feature maps. To alleviate this issue and obtain accurate edges, we propose a hierarchical edge refinement network (HERNet) for accurate saliency detection. In detail, the HERNet is mainly composed of a saliency prediction network and an edge preserving network. Firstly, the saliency prediction network is used to roughly detect the regions of salient objects and is based on a modified U-Net structure. Then, the edge preserving network is used to accurately detect the edges of salient objects, and this network is mainly composed of the atrous spatial pyramid pooling (ASPP) module. Different from the previous indiscriminate supervision strategy, we adopt a new one-to-one hierarchical supervision strategy to supervise the different outputs of the entire network. Experimental results on five traditional benchmark datasets demonstrate that the proposed HERNet performs well when compared with the state-of-the-art methods.

Capturing the grouping and compactness of high-level semantic feature for saliency detection

Saliency detection is an important and challenging research topic due to the variety and complexity of the background and saliency regions. In this paper, we present a novel unsupervised saliency detection approach by exploiting the grouping and compactness characteristics of the high-level semantic features. First, for the high-level semantic feature, the elastic net based hypergraph model is adopted to discover the group structure relationships of salient regional points, and the calculation of the spatial distribution is constructed to detect the compactness of the saliency regions. Next, the grouping-based and compactness-based saliency maps are improved by a propagation algorithm. The propagation process uses an enhanced similarity matrix, which fuses the low-level deep feature and the high-level semantic feature through cross diffusion. Results on four benchmark datasets with pixel-wise accurate labeling demonstrate the effectiveness of the proposed method. Particularly, the proposed unsupervised method achieves competitive performance with deep learning-based methods.

GRID: GRID Resample by Information Distribution

This paper exploits a concise yet efficient initialization strategy to optimize grid sampling-based superpixel segmentation algorithms. Rather than straight distributing all initial seeds evenly, it adopts a context-aware approach to modify their positions and total number via a coarse-to-fine manner. Firstly, half the expected number of seeds are regularly sampled on the image grid, thereby creating a rough distribution of color information for all rectangular cells. A series of fission is then performed on cells that contain excessive color information recursively. In each cell, the local color uniformity is balanced by a dichotomy on one original seed, which generates two new seeds and settles them to spatially symmetrical sub-regions. Therefore, the local concentration of seeds is adaptive to the complexity of regional information. In addition, by calculating the amount of color via a summed area table (SAT), the informative regions can be located at a very low time cost. As a result, superpixels are produced from ideal original seeds with an exact number and exhibit better boundary adherence. Experiments demonstrate that the proposed strategy effectively promotes the performance of simple linear iterative clustering (SLIC) and its variants in terms of several quality measures.

RGB-T Salient Object Detection via Fusing Multi-level CNN Features

RGB-induced salient object detection has recently witnessed substantial progress, which is attributed to the superior feature learning capability of deep convolutional neural networks (CNNs). However, such detections suffer from challenging scenarios characterized by cluttered backgrounds, low-light conditions and variations in illumination. Instead of improving RGB based saliency detection, this paper takes advantage of the complementary benefits of RGB and thermal infrared images. Specifically, we propose a novel end-to-end network for multi-modal salient object detection, which turns the challenge of RGB-T saliency detection to a CNN feature fusion problem. To this end, a backbone network (e.g., VGG-16) is first adopted to extract the coarse features from each RGB or thermal infrared image individually, and then several adjacent-depth feature combination (ADFC) modules are designed to extract multi-level refined features for each single-modal input image, considering that features captured at different depths differ in semantic information and visual details. Subsequently, a multi-branch group fusion (MGF) module is employed to capture the cross-modal features by fusing those features from ADFC modules for a RGB-T image pair at each level. Finally, a joint attention guided bi-directional message passing (JABMP) module undertakes the task of saliency prediction via integrating the multi-level fused features from MGF modules. Experimental results on several public RGB-T salient object detection datasets demonstrate the superiorities of our proposed algorithm over the state-of-the-art approaches, especially under challenging conditions, such as poor illumination, complex background and low contrast.

Local Regression Ranking for Saliency Detection

Saliency detection is an important and challenging research topic due to the variety and complex of the background and saliency regions. In this paper, we present a novel unsupervised saliency detection approach by exploiting a learning-based ranking framework. First, the local linear regression model is adopted to simulate the local manifold structure of every image element, which is approximately linear. Using the background queries from the boundary prior, we construct a unified objective function to globally minimize all the errors of the local models for the whole image element points. The Laplacian matrix is learned via optimizing the unified objective function. Low-level image features as well as high-level semantic information extracted from deep neural networks are used for the Laplacian matrix learning. Based on the learnt Laplacian matrix, the saliency of the image element is measured as the relevance ranking to the background queries. The foreground queries are obtained from the background-based saliency and the relevance ranking to the foreground queries is calculated in the same way as the background-based saliency. Second, we calculate an enhanced similarity matrix by fusing two different-level deep feature metrics through cross diffusion. A propagation algorithm uses this enhanced similarity matrix to better exploit the intrinsic relevance of similar regions and improve the saliency ranking results effectively. Results on four benchmark datasets with pixel-wise accurate labelling demonstrate that the proposed unsupervised method shows better performance compared with the newest state-of-the-art methods and is competitive with deep learning-based methods.

Deep Salient Object Detection with Contextual Information Guidance

Integration of multi-level contextual information, such as feature maps and side outputs, is crucial for Convolutional Neural Networks (CNNs) based salient object detection. However, most existing methods either simply concatenate multi-level feature maps or calculate element-wise addition of multi-level side outputs, thus failing to take full advantages of them. In this work, we propose a new strategy for guiding multi-level contextual information integration, where feature maps and side outputs across layers are fully engaged. Specifically, shallower-level feature maps are guided by the deeper-level side outputs to learn more accurate properties of the salient object. In turn, the deeper-level side outputs can be propagated to high-resolution versions with spatial details complemented by means of shallower-level feature maps. Moreover, a group convolution module is proposed with the aim to achieve high-discriminative feature maps, in which the backbone feature maps are divided into a number of groups and then the convolution is applied to the channels of backbone feature maps within each group. Eventually, the group convolution module is incorporated in the guidance module to further promote the guidance role. Experiments on three public benchmark datasets verify the effectiveness and superiority of the proposed method over the state-of-the-art methods.

Unsupervised Saliency Model with Color Markov Chain for Oil Tank Detection

Traditional oil tank detection methods often use geometric shape information. However, it is difficult to guarantee accurate detection under a variety of disturbance factors, especially various colors, scale differences, and the shadows caused by view angle and illumination. Therefore, we propose an unsupervised saliency model with Color Markov Chain (US-CMC) to deal with oil tank detection. To avoid the influence of shadows, we make use of the CIE Lab space to construct a Color Markov Chain and generate a bottom-up latent saliency map. Moreover, we build a circular feature map based on a radial symmetric circle, which makes true targets to be strengthened for a subjective detection task. Besides, we combine the latent saliency map with the circular feature map, which can effectively suppress other salient regions except for oil tanks. Extensive experimental results demonstrate that it outperforms 15 saliency models for remote sensing images (RSIs). Compared with conventional oil tank detection methods, US-CMC has achieved better results and is also more robust for view angle, shadow, and shape similarity problems.

RGB-'D' Saliency Detection With Pseudo Depth

Recent studies have shown the effectiveness of using depth information in salient object detection. However, the most commonly seen images so far are still RGB images that do not contain the depth data. Meanwhile, the human brain can extract the geometric model of a scene from an RGB-only image and hence provides a 3D perception of the scene. Inspired by this observation, we propose a new concept named RGB-'D' saliency detection, which derives pseudo depth from the RGB images and then performs 3D saliency detection. The pseudo depth can be utilized as image features, prior knowledge, an additional image channel, or independent depth-induced models to boost the performance of traditional RGB saliency models. As an illustration, we develop a new salient object detection algorithm that uses the pseudo depth to derive a depth-driven background prior and a depth contrast feature. Extensive experiments on several standard databases validate the promising performance of the proposed algorithm. In addition, we also adapt two supervised RGB saliency models to our RGB-'D' saliency framework for performance enhancement. The results further demonstrate the generalization ability of the proposed RGB-'D' saliency framework.

Semi-Supervised Salient Object Detection Using a Linear Feedback Control System Model

To overcome the challenging problems in saliency detection, we propose a novel semi-supervised classifier which makes good use of a linear feedback control system (LFCS) model by establishing a relationship between control states and salient object detection. First, we develop a boundary homogeneity model to estimate the initial saliency and background likelihoods, which are regarded as the labeled samples in our semi-supervised learning procedure. Then in order to allocate an optimized saliency value to each superpixel, we present an iterative semi-supervised learning framework which integrates multiple saliency cues and image features using an LFCS model. Via an innovative iteration method, the system gradually converges an optimized stable state, which is associating with an accurate saliency map. This paper also covers comprehensive simulation study based on public datasets, which demonstrates the superiority of the proposed approach.

Salient region detection through salient and non-salient dictionaries

Low-rank representation-based frameworks are becoming popular for the saliency and the object detection because of their easiness and simplicity. These frameworks only need global features to extract the salient objects while the local features are compromised. To deal with this issue, we regularize the low-rank representation through a local graph-regularization and a maximum mean-discrepancy regularization terms. Firstly, we introduce a novel feature space that is extracted by combining the four feature spaces like CIELab, RGB, HOG and LBP. Secondly, we combine a boundary metric, a candidate objectness metric and a candidate distance metric to compute the low-level saliency map. Thirdly, we extract salient and non-salient dictionaries from the low-level saliency. Finally, we regularize the low-rank representation through the Laplacian regularization term that saves the structural and geometrical features and using the mean discrepancy term that reduces the distribution divergence and connections among similar regions. The proposed model is tested against seven latest salient region detection methods using the precision-recall curve, receiver operating characteristics curve, F-measure and mean absolute error. The proposed model remains persistent in all the tests and outperformed against the selected models with higher precision value.

Appearance-Based Salient Regions Detection Using Side-Specific Dictionaries

Image saliency detection is a very helpful step in many computer vision-based smart systems to reduce the computational complexity by only focusing on the salient parts of the image. Currently, the image saliency is detected through representation-based generative schemes, as these schemes are helpful for extracting the concise representations of the stimuli and to capture the high-level semantics in visual information with a small number of active coefficients. In this paper, we propose a novel framework for salient region detection that uses appearance-based and regression-based schemes. The framework segments the image and forms reconstructive dictionaries from four sides of the image. These side-specific dictionaries are further utilized to obtain the saliency maps of the sides. A unified version of these maps is subsequently employed by a representation-based model to obtain a contrast-based salient region map. The map is used to obtain two regression-based maps with LAB and RGB color features that are unified through the optimization-based method to achieve the final saliency map. Furthermore, the side-specific reconstructive dictionaries are extracted from the boundary and the background pixels, which are enriched with geometrical and visual information. The approach has been thoroughly evaluated on five datasets and compared with the seven most recent approaches. The simulation results reveal that our model performs favorably in comparison with the current saliency detection schemes.

Double Low-Rank and Sparse Decomposition for Surface Defect Segmentation of Steel Sheet

Surface defect segmentation supports real-time surface defect detection system of steel sheet by reducing redundant information and highlighting the critical defect regions for high-level image understanding. Existing defect segmentation methods usually lack adaptiveness to different shape, size and scale of the defect object. Based on the observation that the defective area can be regarded as the salient part of image, a saliency detection model using double low-rank and sparse decomposition (DLRSD) is proposed for surface defect segmentation. The proposed method adopts a low-rank assumption which characterizes the defective sub-regions and defect-free background sub-regions respectively. In addition, DLRSD model uses sparse constrains for background sub-regions so as to improve the robustness to noise and uneven illumination simultaneously. Then the Laplacian regularization among spatially adjacent sub-regions is incorporated into the DLRSD model in order to uniformly highlight the defect object. Our proposed DLRSD-based segmentation method consists of three steps: firstly, using DLRSD model to obtain the defect foreground image; then, enhancing the foreground image to establish the good foundation for segmentation; finally, the Otsu’s method is used to choose an optimal threshold automatically for segmentation. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches in terms of both subjective and objective tests. Meanwhile, the proposed method is applicable to industrial detection with limited computational resources.

USEAQ: Ultra-fast Superpixel Extraction via Adaptive Sampling from Quantized Regions

We present a novel and highly efficient superpixel extraction method called USEAQ to generate regular and compact superpixels in an image. To reduce the computational cost of iterative optimization procedures adopted in most recent approaches, the proposed USEAQ for superpixel generation works in a one-pass fashion. It firstly performs joint spatial and color quantizations and groups pixels into regions. It then takes into account the variations between regions, and adaptively samples one or a few superpixel candidates for each region. It finally employs maximum a posteriori (MAP) estimation to assign pixels to the most spatially consistent and perceptually similar superpixels. It turns out that the proposed USEAQ is quite efficient, and the extracted superpixels can precisely adhere to boundaries of objects. Experimental results show that USEAQ achieves better or equivalent performance compared to the stateof- the-art superpixel extraction approaches in terms of boundary recall, undersegmentation error, achievable segmentation accuracy, the average miss rate, average undersegmentation error, and average unexplained variation, and it is significantly faster than these approaches.

Robust Small Target Co-Detection from Airborne Infrared Image Sequences

In this paper, a novel infrared target co-detection model combining the self-correlation features of backgrounds and the commonality features of targets in the spatio-temporal domain is proposed to detect small targets in a sequence of infrared images with complex backgrounds. Firstly, a dense target extraction model based on nonlinear weights is proposed, which can better suppress background of images and enhance small targets than weights of singular values. Secondly, a sparse target extraction model based on entry-wise weighted robust principal component analysis is proposed. The entry-wise weight adaptively incorporates structural prior in terms of local weighted entropy, thus, it can extract real targets accurately and suppress background clutters efficiently. Finally, the commonality of targets in the spatio-temporal domain are used to construct target refinement model for false alarms suppression and target confirmation. Since real targets could appear in both of the dense and sparse reconstruction maps of a single frame, and form trajectories after tracklet association of consecutive frames, the location correlation of the dense and sparse reconstruction maps for a single frame and tracklet association of the location correlation maps for successive frames have strong ability to discriminate between small targets and background clutters. Experimental results demonstrate that the proposed small target co-detection method can not only suppress background clutters effectively, but also detect targets accurately even if with target-like interference.

Toward adaptive fusion of multiple cues for salient region detection

Boundary priors have been extensively studied in salient region detection problems over the past few decades. Although several models based on the boundary prior have achieved good detection performance, there still exist drawbacks. The most common one is that they fail to detect the salient object when the background is complex or the salient object touches the image boundary. In this paper, we propose a novel model to detect the salient region. It is based on background cues and one complementary cue, that is, a foreground cue. A saliency score is obtained via solving an energy optimization problem which takes both the background cue and foreground cue into consideration. Extensive experiments, including both quantitative and qualitative evaluations on five widely used datasets, demonstrate the superiority of our proposed model to several other state-of-the-art models.