Structure-Aware Feature Disentanglement With Knowledge Transfer for Appearance-Changing Place Recognition

Long-term visual place recognition (VPR) is challenging as the environment is subject to drastic appearance changes across different temporal resolutions, such as time of the day, month, and season. A wide variety of existing methods address the problem by means of feature disentangling or image style transfer but ignore the structural information that often remains stable even under environmental condition changes. To overcome this limitation, this article presents a novel structure-aware feature disentanglement network (SFDNet) based on knowledge transfer and adversarial learning. Explicitly, probabilistic knowledge transfer (PKT) is employed to transfer knowledge obtained from the Canny edge detector to the structure encoder. An appearance teacher module is then designed to ensure that the learning of appearance encoder does not only rely on metric learning. The generated content features with structural information are used to measure the similarity of images. We finally evaluate the proposed approach and compare it to state-of-the-art place recognition methods using six datasets with extreme environmental changes. Experimental results demonstrate the effectiveness and improvements achieved using the proposed framework. Source code and some trained models will be available at http://www.tianshu.org.cn.

Learning Condition-Invariant Scene Representations for Place Recognition across the Seasons Using Auto-Encoder and ICA

To localize the position and perfect autonomous navigation, building up a map is essential for mobile robots. The map becomes very important when the weather is not appropriate for the robot. However, the map becomes inconsistent when the robot moves in the environment and detects errors with emotional accuracy. The loop-closure detection is the process through which a robot can acknowledge the location visited previously, which can identify the ultimate solution to the previous problem. The robot faced difficulty identifying its previously visited path when the environment underwent an extreme change. The main motive of our work is to promote a model capable of understanding the scenes that are presented robustly. Moreover, during seasonal changes, this model provides an appropriate loop-closure detection result. Independent component analysis (ICA) and auto-encoder are proposed to complete our research work. ICA is a powerful tool to describe invariant images perfectly. Especially, when the robot moves through a changing environment, ICA provided more accurate outcomes than the other algorithm (baseline algorithm). On the other hand, the auto-encoder can distinguish between two features of scene variant condition and invariant condition. The encoder takes our work’s next steps by discovering possible routes. To analyze the performance, this work uses the baseline method with a precision-recall curve and a fraction of correct matches. The proposed algorithm ICA showed a 91.05% accuracy rate, which is better than the baseline algorithms, and the appropriate route-finding rate using an auto-encoder is also acceptable.

Explicit feature disentanglement for visual place recognition across appearance changes

In the long-term deployment of mobile robots, changing appearance brings challenges for localization. When a robot travels to the same place or restarts from an existing map, global localization is needed, where place recognition provides coarse position information. For visual sensors, changing appearances such as the transition from day to night and seasonal variation can reduce the performance of a visual place recognition system. To address this problem, we propose to learn domain-unrelated features across extreme changing appearance, where a domain denotes a specific appearance condition, such as a season or a kind of weather. We use an adversarial network with two discriminators to disentangle domain-related features and domain-unrelated features from images, and the domain-unrelated features are used as descriptors in place recognition. Provided images from different domains, our network is trained in a self-supervised manner which does not require correspondences between these domains. Besides, our feature extractors are shared among all domains, making it possible to contain more appearance without increasing model complexity. Qualitative and quantitative results on two toy cases are presented to show that our network can disentangle domain-related and domain-unrelated features from given data. Experiments on three public datasets and one proposed dataset for visual place recognition are conducted to illustrate the performance of our method compared with several typical algorithms. Besides, an ablation study is designed to validate the effectiveness of the introduced discriminators in our network. Additionally, we use a four-domain dataset to verify that the network can extend to multiple domains with one model while achieving similar performance.

Condition-Invariant Robot Localization Using Global Sequence Alignment of Deep Features

Localization is one of the essential process in robotics, as it plays an important role in autonomous navigation, simultaneous localization, and mapping for mobile robots. As robots perform large-scale and long-term operations, identifying the same locations in a changing environment has become an important problem. In this paper, we describe a robust visual localization system under severe appearance changes. First, a robust feature extraction method based on a deep variational autoencoder is described to calculate the similarity between images. Then, a global sequence alignment is proposed to find the actual trajectory of the robot. To align sequences, local fragments are detected from the similarity matrix and connected using a rectangle chaining algorithm considering the robot's motion constraint. Since the chained fragments provide reliable clues to find the global path, false matches on featureless structures or partial failures during the alignment could be recovered and perform accurate robot localization in changing environments. The presented experimental results demonstrated the benefits of the proposed method, which outperformed existing algorithms in long-term conditions.

Transforming multiple visual surveys of a natural environment into time-lapses

This article presents a new framework to help transform visual surveys of a natural environment into time-lapses. As data association across year-long variation in appearance continues to represent a formidable challenge, we present success with a map-centric approach, which builds on 3D vision for visual data association. We use a foundation of map point priors and geometric constraints within a dense correspondence image alignment optimization to align images and acquire loop closures between surveys. This framework produces many loop closures between sessions. Outlier loop closures are filtered in the frontend and in the backend to improve robustness. From the result map, the Reprojection Flow algorithm is applied to create time-lapses.

The evaluation of our framework on the Symphony Lake Dataset, which has considerable variation in appearance, led to year-long time-lapses of many different scenes. In comparison with another approach based on using iterative closest point (ICP) plus a homography, our framework produced more and better-quality alignments. With many scenes of the 1.3 km environment consistently aligning well in random image pairs, we next produced 100 time-lapses across 37 surveys captured in a year. Approximately one-third had at least 20 (out of usually 33) well-aligned images, which spanned all four seasons. With promising results, we evaluated the pose error of misaligned image pairs and found that improving map consistency could lead to even better results.