1
|
Qiu H, Zhang X, Wang H, Xiang D, Xiao M, Zhu Z, Wang L. A Robust and Integrated Visual Odometry Framework Exploiting the Optical Flow and Feature Point Method. SENSORS (BASEL, SWITZERLAND) 2023; 23:8655. [PMID: 37896748 PMCID: PMC10611077 DOI: 10.3390/s23208655] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Revised: 10/18/2023] [Accepted: 10/20/2023] [Indexed: 10/29/2023]
Abstract
In this paper, we propose a robust and integrated visual odometry framework exploiting the optical flow and feature point method that achieves faster pose estimate and considerable accuracy and robustness during the odometry process. Our method utilizes optical flow tracking to accelerate the feature point matching process. In the odometry, two visual odometry methods are used: global feature point method and local feature point method. When there is good optical flow tracking and enough key points optical flow tracking matching is successful, the local feature point method utilizes prior information from the optical flow to estimate relative pose transformation information. In cases where there is poor optical flow tracking and only a small number of key points successfully match, the feature point method with a filtering mechanism is used for posing estimation. By coupling and correlating the two aforementioned methods, this visual odometry greatly accelerates the computation time for relative pose estimation. It reduces the computation time of relative pose estimation to 40% of that of the ORB_SLAM3 front-end odometry, while ensuring that it is not too different from the ORB_SLAM3 front-end odometry in terms of accuracy and robustness. The effectiveness of this method was validated and analyzed using the EUROC dataset within the ORB_SLAM3 open-source framework. The experimental results serve as supporting evidence for the efficacy of the proposed approach.
Collapse
Affiliation(s)
- Haiyang Qiu
- School of Naval Architecture and Ocean Engineering, Guangzhou Maritime University, Guangzhou 510725, China; (H.W.); (D.X.); (M.X.)
| | - Xu Zhang
- School of Automation, Jiangsu University of Science and Technology, Zhenjiang 212013, China; (X.Z.); (Z.Z.)
| | - Hui Wang
- School of Naval Architecture and Ocean Engineering, Guangzhou Maritime University, Guangzhou 510725, China; (H.W.); (D.X.); (M.X.)
| | - Dan Xiang
- School of Naval Architecture and Ocean Engineering, Guangzhou Maritime University, Guangzhou 510725, China; (H.W.); (D.X.); (M.X.)
| | - Mingming Xiao
- School of Naval Architecture and Ocean Engineering, Guangzhou Maritime University, Guangzhou 510725, China; (H.W.); (D.X.); (M.X.)
| | - Zhiyu Zhu
- School of Automation, Jiangsu University of Science and Technology, Zhenjiang 212013, China; (X.Z.); (Z.Z.)
| | - Lei Wang
- State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430072, China;
| |
Collapse
|
2
|
Jung JH, Choe Y, Park CG. Photometric Visual-Inertial Navigation With Uncertainty-Aware Ensembles. IEEE T ROBOT 2022. [DOI: 10.1109/tro.2021.3139964] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
3
|
|
4
|
|
5
|
Zhu S, Yang S, Hu P, Qu X. A Robust Optical Flow Tracking Method Based On Prediction Model for Visual-Inertial Odometry. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3079806] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
6
|
|
7
|
GA-Adaptive Template Matching for Offline Shape Motion Tracking Based on Edge Detection: IAS Estimation from the SURVISHNO 2019 Challenge Video for Machine Diagnostics Purposes. ALGORITHMS 2020. [DOI: 10.3390/a13020033] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The estimation of the Instantaneous Angular Speed (IAS) has in recent years attracted a growing interest in the diagnostics of rotating machines. Measurement of the IAS can be used as a source of information of the machine condition per se, or for performing angular resampling through Computed Order Tracking, a practice which is essential to highlight the machine spectral signature in case of non-stationary operational conditions. In these regards, the SURVISHNO 2019 international conference held at INSA Lyon on 8–10 July 2019 proposed a challenge about the estimation of the instantaneous non-stationary speed of a fan from a video taken by a smartphone, a pocket, low-cost device which can nowadays be found in everyone’s pocket. This work originated by the author to produce an offline motion-tracking of the fan (actually, of the head of its locking-screw) and obtaining then a reliable estimate of the IAS. The here proposed algorithm is an update of the established Template Matching (TM) technique (i.e., in the Signal Processing community, a two-dimensional matched filter), which is here integrated into a Genetic Algorithm (GA) search. Using a template reconstructed from a simplified parametric mathematical model of the features of interest (i.e., the known geometry of the edges of the screw head), the GA can be used to adapt the template to match the search image, leading to a hybridization of template-based and feature-based approaches which allows to overcome the well-known issues of the traditional TM related to scaling and rotations of the search image with respect to the template. Furthermore, it is able to resolve the position of the center of the screw head at a resolution that goes beyond the limit of the pixel grid. By repeating the analysis frame after frame and focusing on the angular position of the screw head over time, the proposed algorithm can be used as an effective offline video-tachometer able to estimate the IAS from the video, avoiding the need for expensive high-resolution encoders or tachometers.
Collapse
|
8
|
|
9
|
Volkova A, Gibbens PW. More Robust Features for Adaptive Visual Navigation of UAVs in Mixed Environments. J INTELL ROBOT SYST 2017. [DOI: 10.1007/s10846-017-0650-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
10
|
Forster C, Zhang Z, Gassner M, Werlberger M, Scaramuzza D. SVO: Semidirect Visual Odometry for Monocular and Multicamera Systems. IEEE T ROBOT 2017. [DOI: 10.1109/tro.2016.2623335] [Citation(s) in RCA: 410] [Impact Index Per Article: 58.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
11
|
Alismail H, Kaess M, Browning B, Lucey S. Direct Visual Odometry in Low Light Using Binary Descriptors. IEEE Robot Autom Lett 2017. [DOI: 10.1109/lra.2016.2635686] [Citation(s) in RCA: 38] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
12
|
Cadena C, Carlone L, Carrillo H, Latif Y, Scaramuzza D, Neira J, Reid I, Leonard JJ. Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE T ROBOT 2016. [DOI: 10.1109/tro.2016.2624754] [Citation(s) in RCA: 1565] [Impact Index Per Article: 195.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
13
|
Thanou D, Chou PA, Frossard P. Graph-Based Compression of Dynamic 3D Point Cloud Sequences. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2016; 25:1765-1778. [PMID: 26891486 DOI: 10.1109/tip.2016.2529506] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
This paper addresses the problem of compression of 3D point cloud sequences that are characterized by moving 3D positions and color attributes. As temporally successive point cloud frames share some similarities, motion estimation is key to effective compression of these sequences. It, however, remains a challenging problem as the point cloud frames have varying numbers of points without explicit correspondence information. We represent the time-varying geometry of these sequences with a set of graphs, and consider 3D positions and color attributes of the point clouds as signals on the vertices of the graphs. We then cast motion estimation as a feature-matching problem between successive graphs. The motion is estimated on a sparse set of representative vertices using new spectral graph wavelet descriptors. A dense motion field is eventually interpolated by solving a graph-based regularization problem. The estimated motion is finally used for removing the temporal redundancy in the predictive coding of the 3D positions and the color characteristics of the point cloud sequences. Experimental results demonstrate that our method is able to accurately estimate the motion between consecutive frames. Moreover, motion estimation is shown to bring a significant improvement in terms of the overall compression performance of the sequence. To the best of our knowledge, this is the first paper that exploits both the spatial correlation inside each frame (through the graph) and the temporal correlation between the frames (through the motion estimation) to compress the color and the geometry of 3D point cloud sequences in an efficient way.
Collapse
|
14
|
Autonomous Aerial Refueling Ground Test Demonstration--A Sensor-in-the-Loop, Non-Tracking Method. SENSORS 2015; 15:10948-72. [PMID: 25970254 PMCID: PMC4481912 DOI: 10.3390/s150510948] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/15/2014] [Revised: 04/30/2015] [Accepted: 05/04/2015] [Indexed: 11/23/2022]
Abstract
An essential capability for an unmanned aerial vehicle (UAV) to extend its airborne duration without increasing the size of the aircraft is called the autonomous aerial refueling (AAR). This paper proposes a sensor-in-the-loop, non-tracking method for probe-and-drogue style autonomous aerial refueling tasks by combining sensitivity adjustments of a 3D Flash LIDAR camera with computer vision based image-processing techniques. The method overcomes the inherit ambiguity issues when reconstructing 3D information from traditional 2D images by taking advantage of ready to use 3D point cloud data from the camera, followed by well-established computer vision techniques. These techniques include curve fitting algorithms and outlier removal with the random sample consensus (RANSAC) algorithm to reliably estimate the drogue center in 3D space, as well as to establish the relative position between the probe and the drogue. To demonstrate the feasibility of the proposed method on a real system, a ground navigation robot was designed and fabricated. Results presented in the paper show that using images acquired from a 3D Flash LIDAR camera as real time visual feedback, the ground robot is able to track a moving simulated drogue and continuously narrow the gap between the robot and the target autonomously.
Collapse
|
15
|
Faessler M, Fontana F, Forster C, Mueggler E, Pizzoli M, Scaramuzza D. Autonomous, Vision-based Flight and Live Dense 3D Mapping with a Quadrotor Micro Aerial Vehicle. J FIELD ROBOT 2015. [DOI: 10.1002/rob.21581] [Citation(s) in RCA: 131] [Impact Index Per Article: 14.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Affiliation(s)
- Matthias Faessler
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| | - Flavio Fontana
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| | - Christian Forster
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| | - Elias Mueggler
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| | - Matia Pizzoli
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| | - Davide Scaramuzza
- Robotics and Perception Group; University of Zurich; 8050 Zurich Switzerland
| |
Collapse
|
16
|
Xiong X, Qin K. Linearly estimating all parameters of affine motion using Radon transform. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2014; 23:4311-4321. [PMID: 25069114 DOI: 10.1109/tip.2014.2341932] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
Fast and accurate motion estimation takes an important place in many fields of computer vision and image processing. Using Radon transform to compute projections of the images along specified directions is an effective way to show the relationship between the 2D image object and its projections and estimate motions between the images. All existing projection-based motion estimation methods without the use of iteration have a severe defect that only five of the six affine parameters can be estimated. There are some other methods that can estimate the six parameters, but most of them are usually based on a certain iterative framework, which is computationally intensive and sensitively dependent on the initial values. In this paper, a novel method based on Radon transform is proposed to estimate all the six affine parameters directly. The relationship in the projection domain between a pair of images connected by an affine motion is studied and a linear model is established, by which all the six affine parameters can be directively found. The employment of a hierarchical framework can produce more accurate results. The experimental results reveal that the proposed method has a much better performance than the state-of-the-art methods in this field.
Collapse
|
17
|
Meilland M, Comport AI, Rives P. Dense Omnidirectional RGB-D Mapping of Large-scale Outdoor Environments for Real-time Localization and Autonomous Navigation. J FIELD ROBOT 2014. [DOI: 10.1002/rob.21531] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Affiliation(s)
- Maxime Meilland
- INRIA, I3S/CNRS University of Nice Sophia Antipolis; 06103 Nice France
| | | | | |
Collapse
|
18
|
|
19
|
Martínez C, Mondragón IF, Campoy P, Sánchez-López JL, Olivares-Méndez MA. A Hierarchical Tracking Strategy for Vision-Based Applications On-Board UAVs. J INTELL ROBOT SYST 2013. [DOI: 10.1007/s10846-013-9814-x] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
20
|
Kähler O, Denzler J. Tracking and reconstruction in a combined optimization approach. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2012; 34:387-401. [PMID: 21768650 DOI: 10.1109/tpami.2011.141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/31/2023]
Abstract
We present a novel approach to the structure-from-motion problem which combines the search for correspondences and geometric reconstruction, rather than treating these as separate steps. Through the combination of the two steps, we achieve an implicit feedback of 3D information to aid the correspondence search, and at the same time we avoid an explicit model for tracking errors. The reconstruction results are therefore optimal in case of, for example, Gaussian noise on image intensities. We also present an efficient online framework for structure-from-motion with our combined approach, thoroughly evaluate the method in experiments and compare the results to state-of-the-art methods.
Collapse
Affiliation(s)
- Olaf Kähler
- Department of Engineering Science, University of Oxford, Parks Road, Oxford OX1 3PJ, UK.
| | | |
Collapse
|
21
|
Motion Interchange Patterns for Action Recognition in Unconstrained Videos. COMPUTER VISION – ECCV 2012 2012. [DOI: 10.1007/978-3-642-33783-3_19] [Citation(s) in RCA: 82] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/04/2022]
|
22
|
Silva LS, Scharcanski J. Video segmentation based on motion coherence of particles in a video sequence. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2010; 19:1036-1049. [PMID: 20028629 DOI: 10.1109/tip.2009.2038778] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
This work describes an approach for object-oriented video segmentation based on motion coherence. Using a tracking process based on adaptively sampled points (namely, particles), 2-D motion patterns are identified with an ensemble clustering approach. Particles are clustered to obtain a pixel-wise segmentation in space and time domains. The segmentation result is mapped to an image spatio-temporal feature space. Thus, the different constituent parts of the scene that move coherently along the video sequence are mapped to volumes in this spatio-temporal space. These volumes make the redundancy in the temporal sense more explicit, leading to potential gains in video coding applications. The proposed solution is robust and more generic than similar approaches for 2-D video segmentation found in the literature. In order to illustrate the potential advantages of using the proposed motion segmentation approach in video coding applications, the PSNR of the temporal predictions and the entropies of prediction errors obtained in our experiments are presented, and compared with other methods. Our experiments with real and synthetic sequences suggest that our method also could be used in other image processing and computer vision tasks, besides video coding, such as video information retrieval and video understanding.
Collapse
Affiliation(s)
- Luciano S Silva
- Instituto de Informática, Universidade Federal do Rio Grande do Sul, Porto Alegre, RS, Brazil.
| | | |
Collapse
|
23
|
Abstract
In this paper we describe a new image-based approach to tracking the six-degree-of-freedom trajectory of a stereo camera pair. The proposed technique estimates the pose and subsequently the dense pixel matching between temporal image pairs in a sequence by performing dense spatial matching between images of a stereo reference pair. In this way a minimization approach is employed which directly uses all grayscale information available within the stereo pair (or stereo region) leading to very robust and precise results. Metric 3D structure constraints are imposed by consistently warping corresponding stereo images to generate novel viewpoints at each stereo acquisition. An iterative non-linear trajectory estimation approach is formulated based on a quadrifocal relationship between the image intensities within adjacent views of the stereo pair. A robust M-estimation technique is used to reject outliers corresponding to moving objects within the scene or other outliers such as occlusions and illumination changes. The technique is applied to recovering the trajectory of a moving vehicle in long and difficult sequences of images.
Collapse
Affiliation(s)
- A.I. Comport
- CNRS Laboratoire I3S, 2000 route des Lucioles, Sophia-Antipolis, France,
| | - E. Malis
- INRIA, Sophie-Antipolis Mediterrane 2004 route des Lucioles, Sophia-Antipolis, France,
| | - P. Rives
- INRIA, Sophie-Antipolis Mediterrane 2004 route des Lucioles, Sophia-Antipolis, France,
| |
Collapse
|
24
|
Dedeoglu G, Kanade T, Baker S. The asymmetry of image registration and its application to face tracking. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2007; 29:807-23. [PMID: 17356201 DOI: 10.1109/tpami.2007.1054] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
Most image registration problems are formulated in an asymmetric fashion. Given a pair of images, one is implicitly or explicitly regarded as a template and warped onto the other to match as well as possible. In this paper, we focus on this seemingly arbitrary choice of the roles and reveal how it may lead to biased warp estimates in the presence of relative scaling. We present a principled way of selecting the template and explain why only the correct asymmetric form, with the potential inclusion of a blurring step, can yield an unbiased estimator. We validate our analysis in the domain of model-based face tracking. We show how the usual Active Appearance Model (AAM) formulation overlooks the asymmetry issue, causing the fitting accuracy to degrade quickly when the observed objects are smaller than their model. We formulate a novel, "resolution-aware fitting" (RAF) algorithm that respects the asymmetry and incorporates an explicit model of the blur caused by the camera's sensing elements into the fitting formulation. We compare the RAF algorithm against a state-of-the-art tracker across a variety of resolutions and AAM complexity levels. Experimental results show that RAF significantly improves the estimation accuracy of both shape and appearance parameters when fitting to low-resolution data. Recognizing and accounting for the asymmetry of image registration leads to tangible accuracy improvements in analyzing low-resolution imagery.
Collapse
Affiliation(s)
- Göksel Dedeoglu
- Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.
| | | | | |
Collapse
|
25
|
Riklin-Raviv T, Kiryati N, Sochen N. Prior-based Segmentation and Shape Registration in the Presence of Perspective Distortion. Int J Comput Vis 2006. [DOI: 10.1007/s11263-006-9042-y] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|
26
|
A Feature-based Approach for Dense Segmentation and Estimation of Large Disparity Motion. Int J Comput Vis 2006. [DOI: 10.1007/s11263-006-6660-3] [Citation(s) in RCA: 14] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|
27
|
|
28
|
Wills J, Belongie S. A Feature-Based Approach for Determining Dense Long Range Correspondences. LECTURE NOTES IN COMPUTER SCIENCE 2004. [DOI: 10.1007/978-3-540-24672-5_14] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
|
29
|
Riklin-Raviv T, Kiryati N, Sochen N. Unlevel-Sets: Geometry and Prior-Based Segmentation. ACTA ACUST UNITED AC 2004. [DOI: 10.1007/978-3-540-24673-2_5] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/20/2023]
|
30
|
On Affine Invariant Clustering and Automatic Cast Listing in Movies. COMPUTER VISION — ECCV 2002 2002. [DOI: 10.1007/3-540-47977-5_20] [Citation(s) in RCA: 32] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
|
31
|
Torr PHS, Zisserman A. Feature Based Methods for Structure and Motion Estimation. VISION ALGORITHMS: THEORY AND PRACTICE 2000. [DOI: 10.1007/3-540-44480-7_19] [Citation(s) in RCA: 73] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
|