NeuroBayesSLAM: Neurobiologically inspired Bayesian integration of multisensory information for robot navigation

Semantic discrete decoder based on adaptive pixel clustering for monocular depth estimation

Monocular depth estimation (MDE) has long been a popular and challenging task. Currently, mainstream methods mainly include regression methods based on geometric constraints and ordinal regression methods based on discretized depth intervals. However, they both overlook the fact that depth values within objects often exhibit some degree of continuity, while depth values between objects exhibit varying degrees of discontinuity. Based on this, we propose a more general approach to monocular depth estimation called APCDepth. This method does not treat MDE as an ordinal regression task but rather as a continuous regression task to ensure the continuity of depth values within objects. To focus on the discontinuity of depth values between objects, we propose an Adaptive Pixel Clustering (APC) module to semantically discretize encoder deep features, and align the discretized feature maps to a larger resolution using our proposed Cross-Semantic Alignment (CSA) module. Additionally, to tackle the quadratic complexity issue introduced by Transformers as decoders in depth estimation, we propose a Deformable Feature Pyramid Network (DeFPN) with sparse attention for multi-scale feature fusion. Furthermore, experimental results on the KITTI and NYU datasets validate the effectiveness of APCDepth and demonstrate outstanding performance.

A Review of Brain-Inspired Cognition and Navigation Technology for Mobile Robots

Brain-inspired navigation technologies combine environmental perception, spatial cognition, and target navigation to create a comprehensive navigation research system. Researchers have used various sensors to gather environmental data and enhance environmental perception using multimodal information fusion. In spatial cognition, a neural network model is used to simulate the navigation mechanism of the animal brain and to construct an environmental cognition map. However, existing models face challenges in achieving high navigation success rate and efficiency. In addition, the limited incorporation of navigation mechanisms borrowed from animal brains necessitates further exploration. On the basis of the brain-inspired navigation process, this paper launched a systematic study on brain-inspired environment perception, brain-inspired spatial cognition, and goal-based navigation in brain-inspired navigation, which provides a new classification of brain-inspired cognition and navigation techniques and a theoretical basis for subsequent experimental studies. In the future, brain-inspired navigation technology should learn from more perfect brain-inspired mechanisms to improve its generalization ability and be simultaneously applied to large-scale distributed intelligent body cluster navigation. The multidisciplinary nature of brain-inspired navigation technology presents challenges, and multidisciplinary scholars must cooperate to promote the development of this technology.

Multi-level feature fusion and joint refinement for simultaneous object pose estimation and camera localization

Object pose estimation and camera localization are critical in various applications. However, achieving algorithm universality, which refers to category-level pose estimation and scene-independent camera localization, presents challenges for both techniques. Although the two tasks keep close relationships due to spatial geometry constraints, different tasks require distinct feature extractions. This paper pays attention to a unified RGB-D based framework that simultaneously performs category-level object pose estimation and scene-independent camera localization. The framework consists of a pose estimation branch called SLO-ObjNet, a localization branch called SLO-LocNet, a pose confidence calculation process and object-level optimization. At the start, we obtain the initial camera and object results from SLO-LocNet and SLO-ObjNet. In these two networks, we design there-level feature fusion modules as well as the loss function to achieve feature sharing between two tasks. Then the proposed approach involves a confidence calculation process to determine the accuracy of object poses obtained. Additionally, an object-level Bundle Adjustment (BA) optimization algorithm is further used to improve the precision of these techniques. The BA algorithm establishes relationships among feature points, objects, and cameras with the usage of camera-point, camera-object, and object-point metrics. To evaluate the performance of this approach, experiments are conducted on localization and pose estimation datasets including REAL275, CAMERA25, LineMOD, YCB-Video, 7 Scenes, ScanNet and TUM RGB-D. The results show that this approach outperforms existing methods in terms of both estimation and localization accuracy. Additionally, SLO-LocNet and SLO-ObjNet are trained on ScanNet data and tested on 7 Scenes and TUM RGB-D datasets to demonstrate its universality performance. Finally, we also highlight the positive effects of fusion modules, loss function, confidence process and BA for improving overall performance.

Joint estimation of pose, depth, and optical flow with a competition-cooperation transformer network

Estimating depth, ego-motion, and optical flow from consecutive frames is a critical task in robot navigation and has received significant attention in recent years. In this study, we propose PDF-Former, an unsupervised joint estimation network comprising a full transformer-based framework, as well as a competition and cooperation mechanism. The transformer framework captures global feature dependencies and is customized for different task types, thereby improving the performance of sequential tasks. The competition and cooperation mechanisms enable the network to obtain additional supervisory information at different training stages. Specifically, the competition mechanism is implemented early in training to achieve iterative optimization of 6 DOF poses (rotation and translation information from the target image to the two reference images), the depth of target image, and optical flow (from the target image to the two reference images) estimation in a competitive manner. In contrast, the cooperation mechanism is implemented later in training to facilitate the transmission of results among the three networks and mutually optimize the estimation results. We conducted experiments on the KITTI dataset, and the results indicate that PDF-Former has significant potential to enhance the accuracy and robustness of sequential tasks in robot navigation.

Statistically Optimal Cue Integration During Human Spatial Navigation

In 2007, Cheng and colleagues published their influential review wherein they analyzed the literature on spatial cue interaction during navigation through a Bayesian lens, and concluded that models of optimal cue integration often applied in psychophysical studies could explain cue interaction during navigation. Since then, numerous empirical investigations have been conducted to assess the degree to which human navigators are optimal when integrating multiple spatial cues during a variety of navigation-related tasks. In the current review, we discuss the literature on human cue integration during navigation that has been published since Cheng et al.'s original review. Evidence from most studies demonstrate optimal navigation behavior when humans are presented with multiple spatial cues. However, applications of optimal cue integration models vary in their underlying assumptions (e.g., uninformative priors and decision rules). Furthermore, cue integration behavior depends in part on the nature of the cues being integrated and the navigational task (e.g., homing versus non-home goal localization). We discuss the implications of these models and suggest directions for future research.

Brain-like position measurement method based on improved optical flow algorithm

In this paper, a brain-like navigation scheme based on fuzzy kernel C-means (FKCM) clustering assisted pyramid Lucas Kanade (LK) optical flow algorithm is developed to measure the position of vehicle. The Speed Cell and Place Cell in animals' brain are introduced to construct the brain-like navigation mechanism which involves the optical flow method and image template matching to imitate the cells above-mentioned separately. To eliminate the singular values during optical flow calculation, the output of pyramid LK algorithm is clustered by FKCM algorithm firstly. Then, the velocity is calculated and integrated to get the position of the vehicle, and the brain-like navigation scheme is introduced to correct the position measurement errors by eliminating the accumulated errors resulting from velocity integration. The prominent advantages of the presented method are: (i) a pure visual brain-like position measurement method based on the concept of speed cells and place cells is proposed, making visual navigation more accurate and intelligent; (ii) the FKCM algorithm is used to eliminate the singular value of the pyramid LK algorithm, which improves the calculated velocity accuracy. Also, experimental comparison with classical pyramid LK algorithm is given to illustrate the superiority of the proposed method in position measurement.

A biologically inspired auto-associative network with sparse temporal population coding

Associative system has attracted increasing attention for it can store basic information and then infer details to match perception with an efficient self-organization algorithm. However, the implementation of the associative system with the application of real-world data is relatively difficult. To address this issue, we propose a novel biologically inspired auto-associative (BIAA) network to explore the structure, encoding and formation of associative memory as well as to extend the ability to real-world application. Our network is constructed by imitating the organization of the cortical minicolumns where each minicolumn contains plenty of parallel biological spiking neurons. To allow the network to learn and predict one symbol per theta cycle, we incorporate synaptic delay and theta oscillation into the neuron dynamic process. Subsequently, we design a sparse temporal population (STP) coding scheme that allows each input symbol to be represented as stable, unique, and easily recallable sparsely distributed representations. By combining associative learning dynamics with the STP coding, our network realizes efficient storage and inference in an ordered manner. Experimental results indicate that the proposed network successfully performs sequence retrieval from partial text and sequence recovery from distorted information. BIAA network provides new insight into introducing biologically inspired mechanisms into associative system and has enormous potential for hardware and software applications.

Predictive hierarchical reinforcement learning for path-efficient mapless navigation with moving target

Deep reinforcement learning (DRL) has been proven as a powerful approach for robot navigation over the past few years. DRL-based navigation does not require the pre-construction of a map, instead, high-performance navigation skills can be learned from trial-and-error experiences. However, recent DRL-based approaches mostly focus on a fixed navigation target. It is noted that when navigating to a moving target without maps, the performance of the standard RL structure drops dramatically on both the success rate and path efficiency. To address the mapless navigation problem with moving target, the predictive hierarchical DRL (pH-DRL) framework is proposed by integrating the long-term trajectory prediction to provide a cost-effective solution. In the proposed framework, the lower-level policy of the RL agent learns robot control actions to a specified goal, and the higher-level policy learns to make long-range planning of shorter navigation routes by sufficiently exploiting the predicted trajectories. By means of making decisions over two level of policies, the pH-DRL framework is robust to the unavoidable errors in long-term predictions. With the application of deep deterministic policy gradient (DDPG) for policy optimization, the pH-DDPG algorithm is developed based on the pH-DRL structure. Finally, through comparative experiments on the Gazebo simulator with several variants of the DDPG algorithm, the results demonstrate that the pH-DDPG outperforms other algorithms and achieves a high success rate and efficiency even though the target moves fast and randomly.

Single-trial motor imagery electroencephalogram intention recognition by optimal discriminant hyperplane and interpretable discriminative rectangle mixture model

Spatial filtering is widely used in brain-computer interface (BCI) systems to augmented signal characteristics of electroencephalogram (EEG) signals. In this study, a spatial domain filtering based EEG feature extraction method, optimal discriminant hyperplane-common spatial subspace decomposition (ODH-CSSD) is proposed. Specifically, the multi-dimensional EEG features were extracted from the original EEG signals by common space subspace decomposition (CSSD) algorithm, and the optimal feature criterion was established to find the multi-dimensional optimal projection space. A classic method of data dimension optimizing is using the eigenvectors of a lumped covariance matrix corresponding to the maximum eigenvalues. Then, the cost function is defined as the extreme value of the discriminant criterion, and the orthogonal N discriminant vectors corresponding to the N extreme value of the criterion are solved and constructed into the N-dimensional optimal feature space. Finally, the multi-dimensional EEG features are projected into the N-dimensional optimal projection space to obtain the optimal N-dimensional EEG features. Moreover, this study involves the extraction of two-dimensional and three-dimensional optimal EEG features from motor imagery EEG datasets, and the optimal EEG features are identified using the interpretable discriminative rectangular mixture model (DRMM). Experimental results show that the accuracy of DRMM to identify two-dimensional optimal features is more than 0.91, and the highest accuracy even reaches 0.975. Meanwhile, DRMM has the most stable recognition accuracy for two-dimensional optimal features, and its average clustering accuracy reaches 0.942, the gap between the accuracy of the DRMM with the accuracy of the FCM and K-means can reach 0.26. And the optimal three-dimensional features, for most subjects, the clustering accuracy of DRMM is higher than that of FCM and K-means. In general, the decision rectangle obtained by DRMM can clearly explain the difference of each cluster, notably, the optimization of multidimensional EEG features by optimal projection is superior to Fisher's ratio, and this method provides an alternative for the application of BCI.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-021-09768-w.

Toward Cognitive Navigation: Design and Implementation of a Biologically Inspired Head Direction Cell Network

As a vital cognitive function of animals, the navigation skill is first built on the accurate perception of the directional heading in the environment. Head direction cells (HDCs), found in the limbic system of animals, are proven to play an important role in identifying the directional heading allocentrically in the horizontal plane, independent of the animal's location and the ambient conditions of the environment. However, practical HDC models that can be implemented in robotic applications are rarely investigated, especially those that are biologically plausible and yet applicable to the real world. In this article, we propose a computational HDC network that is consistent with several neurophysiological findings concerning biological HDCs and then implement it in robotic navigation tasks. The HDC network keeps a representation of the directional heading only relying on the angular velocity as an input. We examine the proposed HDC model in extensive simulations and real-world experiments and demonstrate its excellent performance in terms of accuracy and real-time capability.

Entorhinal-hippocampal interactions lead to globally coherent representations of space

The firing maps of grid cells in the entorhinal cortex are thought to provide an efficient metric system capable of supporting spatial inference in all environments. However, whether spatial representations of grid cells are determined by local environment cues or are organized into globally coherent patterns remains undetermined. We propose a navigation model containing a path integration system in the entorhinal cortex and a cognitive map system in the hippocampus. In the path integration system, grid cell network and head direction (HD) cell network integrate movement and visual information, and form attractor states to represent the positions and head directions of the animal. In the cognitive map system, a topological map is constructed capturing the attractor states of the path integration system as nodes and the transitions between attractor states as links. On loop closure, when the animal revisits a familiar place, the topological map is calibrated to minimize odometry errors. The change of the topological map is mapped back to the path integration system, to correct the states of the grid cells and the HD cells. The proposed model was tested on iRat, a rat-like miniature robot, in a realistic maze. Experimental results showed that, after familiarization of the environment, both grid cells and HD cells develop globally coherent firing maps by map calibration and activity correction. These results demonstrate that the hippocampus and the entorhinal cortex work together to form globally coherent metric representations of the environment. The underlying mechanisms of the hippocampal-entorhinal circuit in capturing the structure of the environment from sequences of experience are critical for understanding episodic memory.

Bayesian models of human navigation behaviour in an augmented reality audiomaze

We investigated Bayesian modelling of human whole‐body motion capture data recorded during an exploratory real‐space navigation task in an “Audiomaze” environment (see the companion paper by Miyakoshi et al. in the same volume) to study the effect of map learning on navigation behaviour. There were three models, a feedback‐only model (no map learning), a map resetting model (single‐trial limited map learning), and a map updating model (map learning accumulated across three trials). The estimated behavioural variables included step sizes and turning angles. Results showed that the estimated step sizes were constantly more accurate using the map learning models than the feedback‐only model. The same effect was confirmed for turning angle estimates, but only for data from the third trial. We interpreted these results as Bayesian evidence of human map learning on navigation behaviour. Furthermore, separating the participants into groups of egocentric and allocentric navigators revealed an advantage for the map updating model in estimating step sizes, but only for the allocentric navigators. This interaction indicated that the allocentric navigators may take more advantage of map learning than do egocentric navigators. We discuss relationships of these results to simultaneous localization and mapping (SLAM) problem.

A brain-inspired compact cognitive mapping system

In many simultaneous localization and mapping (SLAM) systems, the map of the environment grows over time as the robot explores the environment. The ever-growing map prevents long-term mapping, especially in large-scale environments. In this paper, we develop a compact cognitive mapping approach inspired by neurobiological experiments. Mimicking the firing activities of neighborhood cells, neighborhood fields determined by movement information, i.e. translation and rotation, are modeled to describe one of the distinct segments of the explored environment. The vertices with low neighborhood field activities are avoided to be added into the cognitive map. The optimization of the cognitive map is formulated as a robust non-linear least squares problem constrained by the transitions between vertices, and is numerically solved efficiently. According to the cognitive decision-making of place familiarity, loop closure edges are clustered depending on time intervals, and then batch global optimization of the cognitive map is performed to satisfy the combined constraint of the whole cluster. After the loop closure process, scene integration is performed, in which revisited vertices are removed subsequently to further reduce the size of the cognitive map. The compact cognitive mapping approach is tested on a monocular visual SLAM system in a naturalistic maze for a biomimetic animated robot. Our results demonstrate that the proposed method largely restricts the growth of the size of the cognitive map over time, and meanwhile, the compact cognitive map correctly represents the overall layout of the environment. The compact cognitive mapping method is well suitable for the representation of large-scale environments to achieve long-term robot navigation.

Video data for the cognitive mapping process of NeuroBayesSLAM system

Simultaneous localization and mapping (SLAM), which addresses the problem of constructing a spatial map of an unknown environment while simultaneously determining the mobile robot's position relative to this map, is regarded as one of the key technologies in mobile robot navigation. This data article presents four raw video files, demonstrating the mapping and localization processes of NeuroBayesSLAM, a neurobiologically inspired SLAM system, on two publicly available datasets, namely the St Lucia suburb dataset and the iRat Australia dataset. The cognitive mapping process was recorded by a free screen recorder software on ubuntu Linux system. Neural activities of the head-direction cells and the grid cells, the local view templates of visual scenes, and experience map were included. These data envision the possibility of transferring the multisensory integration mechanism found in the spatial memory circuits of the mammalian brain to develop intelligent cognitive mapping systems for indoor and large outdoor environments as in the research article “NeuroBayesSLAM: Neurobiologically Inspired Bayesian Integration of Multisensory Information for Robot Navigation” Zeng et al., 2020.