1
|
Gionfrida L, Kim D, Scaramuzza D, Farina D, Howe RD. Wearable robots for the real world need vision. Sci Robot 2024; 9:eadj8812. [PMID: 38776377 DOI: 10.1126/scirobotics.adj8812] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2023] [Accepted: 04/24/2024] [Indexed: 05/25/2024]
Abstract
To enhance wearable robots, understanding user intent and environmental perception with novel vision approaches is needed.
Collapse
Affiliation(s)
- Letizia Gionfrida
- Department of Informatics, Faculty of Natural Mathematics and Engineering Sciences, King's College London, Bush House, 30 Aldwych, London WC2B 4BG, UK
- John A. Paulson School of Engineering and Applied Sciences and Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, MA, USA
| | - Daekyum Kim
- School of Smart Mobility, Korea University, Seoul 02841, South Korea
- School of Mechanical Engineering, Korea University, Seoul 02841, South Korea
| | - Davide Scaramuzza
- Robotics and Perception Group, Department of Informatics, University of Zurich, Andreasstrasse 15, 8050 Zurich, Switzerland
| | - Dario Farina
- Department of Bioengineering, Faculty of Engineering, Imperial College London, Exhibition Rd, South Kensington, London SW7 2BX, UK
| | - Robert D Howe
- John A. Paulson School of Engineering and Applied Sciences and Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, MA, USA
| |
Collapse
|
2
|
Yang Y, Zhu F, Zhang X, Chen P, Wang Y, Zhu J, Ding Y, Cheng L, Li C, Jiang H, Wang Z, Lin P, Shi T, Wang M, Liu Q, Xu N, Liu M. Firing feature-driven neural circuits with scalable memristive neurons for robotic obstacle avoidance. Nat Commun 2024; 15:4318. [PMID: 38773067 PMCID: PMC11109161 DOI: 10.1038/s41467-024-48399-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 04/30/2024] [Indexed: 05/23/2024] Open
Abstract
Neural circuits with specific structures and diverse neuronal firing features are the foundation for supporting intelligent tasks in biology and are regarded as the driver for catalyzing next-generation artificial intelligence. Emulating neural circuits in hardware underpins engineering highly efficient neuromorphic chips, however, implementing a firing features-driven functional neural circuit is still an open question. In this work, inspired by avoidance neural circuits of crickets, we construct a spiking feature-driven sensorimotor control neural circuit consisting of three memristive Hodgkin-Huxley neurons. The ascending neurons exhibit mixed tonic spiking and bursting features, which are used for encoding sensing input. Additionally, we innovatively introduce a selective communication scheme in biology to decode mixed firing features using two descending neurons. We proceed to integrate such a neural circuit with a robot for avoidance control and achieve lower latency than conventional platforms. These results provide a foundation for implementing real brain-like systems driven by firing features with memristive neurons and put constructing high-order intelligent machines on the agenda.
Collapse
Affiliation(s)
- Yue Yang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Fangduo Zhu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Xumeng Zhang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China.
| | - Pei Chen
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Yongzhou Wang
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Jiaxue Zhu
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Yanting Ding
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Lingli Cheng
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Chao Li
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Hao Jiang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Zhongrui Wang
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, 999077, China
| | - Peng Lin
- College of Computer Science and Technology, Zhejiang University, Zhejiang, 310027, China
| | - Tuo Shi
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| | - Ming Wang
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Qi Liu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China.
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China.
| | - Ningsheng Xu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
| | - Ming Liu
- State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, 200433, China
- Key Laboratory of Microelectronics Device & Integrated Technology, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China
| |
Collapse
|
3
|
Liu DM, Cui JS, Zhong YJ, Min CW, Zhang FR, Feng DZ. A fast and high precision multi-robot environment modeling based on M-BFSI: Bidirectional filtering and scene identification method. iScience 2024; 27:109721. [PMID: 38706853 PMCID: PMC11068629 DOI: 10.1016/j.isci.2024.109721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 03/20/2024] [Accepted: 04/08/2024] [Indexed: 05/07/2024] Open
Abstract
This article designs and implements a fast and high-precision multi-robot environment modeling method based on bidirectional filtering and scene identification. To solve the problem of feature tracking failure caused by large angle rotation, a bidirectional filtering mechanism is introduced to improve the error-matching elimination algorithm. A global key frame database for multiple robots is proposed based on a pretraining dictionary to convert images into a bag of words vectors. The images captured by different sub-robots are compared with the database for similarity score calculation, so as to realize fast identification and search of similar scenes. The coordinate transformation from local map to global map and the cooperative SLAM exploration of multiple robots is completed by the best matching image and the transformation matrix. The experimental results show that the proposed algorithm can effectively close the predicted trajectory of the sub-robot, thus achieving high-precision collaborative environment modeling.
Collapse
Affiliation(s)
- Dai-ming Liu
- School of Aerospace Science and Technology, Xidian University, Xi’an 710126, China
| | - Jia-shan Cui
- School of Aerospace Science and Technology, Xidian University, Xi’an 710126, China
| | - Yong-jian Zhong
- Shanghai Electro-Mechanical Engineering Institute, Shanghai 201109, China
| | - Chang-wan Min
- School of Aerospace Science and Technology, Xidian University, Xi’an 710126, China
| | - Fang-rui Zhang
- China Academy of Space Technology, Xian Branch, Xi’an 710100, China
| | - Dong-zhu Feng
- School of Aerospace Science and Technology, Xidian University, Xi’an 710126, China
| |
Collapse
|
4
|
Zhang L, Deng J. Deep Compressed Communication and Application in Multi-Robot 2D-Lidar SLAM: An Intelligent Huffman Algorithm. SENSORS (BASEL, SWITZERLAND) 2024; 24:3154. [PMID: 38794008 PMCID: PMC11124910 DOI: 10.3390/s24103154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/07/2024] [Revised: 05/09/2024] [Accepted: 05/14/2024] [Indexed: 05/26/2024]
Abstract
Multi-robot Simultaneous Localization and Mapping (SLAM) systems employing 2D lidar scans are effective for exploration and navigation within GNSS-limited environments. However, scalability concerns arise with larger environments and increased robot numbers, as 2D mapping necessitates substantial processor memory and inter-robot communication bandwidth. Thus, data compression prior to transmission becomes imperative. This study investigates the problem of communication-efficient multi-robot SLAM based on 2D maps and introduces an architecture that enables compressed communication, facilitating the transmission of full maps with significantly reduced bandwidth. We propose a framework employing a lightweight feature extraction Convolutional Neural Network (CNN) for a full map, followed by an encoder combining Huffman and Run-Length Encoding (RLE) algorithms to further compress a full map. Subsequently, a lightweight recovery CNN was designed to restore map features. Experimental validation involves applying our compressed communication framework to a two-robot SLAM system. The results demonstrate that our approach reduces communication overhead by 99% while maintaining map quality. This compressed communication strategy effectively addresses bandwidth constraints in multi-robot SLAM scenarios, offering a practical solution for collaborative SLAM applications.
Collapse
Affiliation(s)
- Liang Zhang
- School of Electrical Engineering and Automation, Anhui University, Hefei 230093, China
| | | |
Collapse
|
5
|
Richter A, Steinmann T, Rosenthal JC, Rupitsch SJ. Advances in Real-Time 3D Reconstruction for Medical Endoscopy. J Imaging 2024; 10:120. [PMID: 38786574 PMCID: PMC11122342 DOI: 10.3390/jimaging10050120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Revised: 04/23/2024] [Accepted: 04/24/2024] [Indexed: 05/25/2024] Open
Abstract
This contribution is intended to provide researchers with a comprehensive overview of the current state-of-the-art concerning real-time 3D reconstruction methods suitable for medical endoscopy. Over the past decade, there have been various technological advancements in computational power and an increased research effort in many computer vision fields such as autonomous driving, robotics, and unmanned aerial vehicles. Some of these advancements can also be adapted to the field of medical endoscopy while coping with challenges such as featureless surfaces, varying lighting conditions, and deformable structures. To provide a comprehensive overview, a logical division of monocular, binocular, trinocular, and multiocular methods is performed and also active and passive methods are distinguished. Within these categories, we consider both flexible and non-flexible endoscopes to cover the state-of-the-art as fully as possible. The relevant error metrics to compare the publications presented here are discussed, and the choice of when to choose a GPU rather than an FPGA for camera-based 3D reconstruction is debated. We elaborate on the good practice of using datasets and provide a direct comparison of the presented work. It is important to note that in addition to medical publications, publications evaluated on the KITTI and Middlebury datasets are also considered to include related methods that may be suited for medical 3D reconstruction.
Collapse
Affiliation(s)
- Alexander Richter
- Fraunhofer Institute for High-Speed Dynamics, Ernst–Mach–Institut (EMI), Ernst-Zermelo-Straße 4, 79104 Freiburg, Germany
- Electrical Instrumentation and Embedded Systems, Albert–Ludwigs–Universität Freiburg, Goerges-Köhler-Allee 106, 79110 Freiburg, Germany; (T.S.); (S.J.R.)
| | - Till Steinmann
- Electrical Instrumentation and Embedded Systems, Albert–Ludwigs–Universität Freiburg, Goerges-Köhler-Allee 106, 79110 Freiburg, Germany; (T.S.); (S.J.R.)
| | - Jean-Claude Rosenthal
- Fraunhofer Institute for Telecommunications, Heinrich–Hertz–Institut (HHI), Einsteinufer 37, 10587 Berlin, Germany
| | - Stefan J. Rupitsch
- Electrical Instrumentation and Embedded Systems, Albert–Ludwigs–Universität Freiburg, Goerges-Köhler-Allee 106, 79110 Freiburg, Germany; (T.S.); (S.J.R.)
| |
Collapse
|
6
|
Zhuang L, Zhong X, Xu L, Tian C, Yu W. Visual SLAM for Unmanned Aerial Vehicles: Localization and Perception. SENSORS (BASEL, SWITZERLAND) 2024; 24:2980. [PMID: 38793834 PMCID: PMC11126069 DOI: 10.3390/s24102980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 05/01/2024] [Accepted: 05/04/2024] [Indexed: 05/26/2024]
Abstract
Localization and perception play an important role as the basis of autonomous Unmanned Aerial Vehicle (UAV) applications, providing the internal state of movements and the external understanding of environments. Simultaneous Localization And Mapping (SLAM), one of the critical techniques for localization and perception, is facing technical upgrading, due to the development of embedded hardware, multi-sensor technology, and artificial intelligence. This survey aims at the development of visual SLAM and the basis of UAV applications. The solutions to critical problems for visual SLAM are shown by reviewing state-of-the-art and newly presented algorithms, providing the research progression and direction in three essential aspects: real-time performance, texture-less environments, and dynamic environments. Visual-inertial fusion and learning-based enhancement are discussed for UAV localization and perception to illustrate their role in UAV applications. Subsequently, the trend of UAV localization and perception is shown. The algorithm components, camera configuration, and data processing methods are also introduced to give comprehensive preliminaries. In this paper, we provide coverage of visual SLAM and its related technologies over the past decade, with a specific focus on their applications in autonomous UAV applications. We summarize the current research, reveal potential problems, and outline future trends from academic and engineering perspectives.
Collapse
Affiliation(s)
- Licong Zhuang
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Xiaorong Zhong
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Linjie Xu
- The College of Civil and Transportation Engineering, Shenzhen University, 3688 Nanhai Avenue, Nanshan District, Shenzhen 518060, China;
| | - Chunbao Tian
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Yutang Street, Guangming District, Shenzhen 518132, China; (L.Z.); (X.Z.); (C.T.)
| | - Wenshuai Yu
- The College of Civil and Transportation Engineering, Shenzhen University, 3688 Nanhai Avenue, Nanshan District, Shenzhen 518060, China;
| |
Collapse
|
7
|
Lin S, Chen X, Xiao G, Wang H, Huang F, Weng J. Multi-Stage Network With Geometric Semantic Attention for Two-View Correspondence Learning. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:3031-3046. [PMID: 38656841 DOI: 10.1109/tip.2024.3391002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
The removal of outliers is crucial for establishing correspondence between two images. However, when the proportion of outliers reaches nearly 90%, the task becomes highly challenging. Existing methods face limitations in effectively utilizing geometric transformation consistency (GTC) information and incorporating geometric semantic neighboring information. To address these challenges, we propose a Multi-Stage Geometric Semantic Attention (MSGSA) network. The MSGSA network consists of three key modules: the multi-branch (MB) module, the GTC module, and the geometric semantic attention (GSA) module. The MB module, structured with a multi-branch design, facilitates diverse and robust spatial transformations. The GTC module captures transformation consistency information from the preceding stage. The GSA module categorizes input based on the prior stage's output, enabling efficient extraction of geometric semantic information through a graph-based representation and inter-category information interaction using Transformer. Extensive experiments on the YFCC100M and SUN3D datasets demonstrate that MSGSA outperforms current state-of-the-art methods in outlier removal and camera pose estimation, particularly in scenarios with a high prevalence of outliers. Source code is available at https://github.com/shuyuanlin.
Collapse
|
8
|
Okabe R, Xue S, Vavrek JR, Yu J, Pavlovsky R, Negut V, Quiter BJ, Cates JW, Liu T, Forget B, Jegelka S, Kohse G, Hu LW, Li M. Tetris-inspired detector with neural network for radiation mapping. Nat Commun 2024; 15:3061. [PMID: 38594238 PMCID: PMC11004156 DOI: 10.1038/s41467-024-47338-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Accepted: 03/27/2024] [Indexed: 04/11/2024] Open
Abstract
Radiation mapping has attracted widespread research attention and increased public concerns on environmental monitoring. Regarding materials and their configurations, radiation detectors have been developed to identify the position and strength of the radioactive sources. However, due to the complex mechanisms of radiation-matter interaction and data limitation, high-performance and low-cost radiation mapping is still challenging. Here, we present a radiation mapping framework using Tetris-inspired detector pixels. Applying inter-pixel padding for enhancing contrast between pixels and neural networks trained with Monte Carlo (MC) simulation data, a detector with as few as four pixels can achieve high-resolution directional prediction. A moving detector with Maximum a Posteriori (MAP) further achieved radiation position localization. Field testing with a simple detector has verified the capability of the MAP method for source localization. Our framework offers an avenue for high-quality radiation mapping with simple detector configurations and is anticipated to be deployed for real-world radiation detection.
Collapse
Affiliation(s)
- Ryotaro Okabe
- Quantum Measurement Group, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
- Department of Chemistry, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Shangjie Xue
- Quantum Measurement Group, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Jayson R Vavrek
- Applied Nuclear Physics Program, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
| | - Jiankai Yu
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Ryan Pavlovsky
- Applied Nuclear Physics Program, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
| | - Victor Negut
- Applied Nuclear Physics Program, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
| | - Brian J Quiter
- Applied Nuclear Physics Program, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
| | - Joshua W Cates
- Applied Nuclear Physics Program, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
| | - Tongtong Liu
- Quantum Measurement Group, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Department of Physics, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Benoit Forget
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Stefanie Jegelka
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Gordon Kohse
- Nuclear Reactor Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Lin-Wen Hu
- Nuclear Reactor Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Mingda Li
- Quantum Measurement Group, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| |
Collapse
|
9
|
Cai Y, Ou Y, Qin T. Improving SLAM Techniques with Integrated Multi-Sensor Fusion for 3D Reconstruction. SENSORS (BASEL, SWITZERLAND) 2024; 24:2033. [PMID: 38610245 PMCID: PMC11014387 DOI: 10.3390/s24072033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/03/2024] [Revised: 03/09/2024] [Accepted: 03/18/2024] [Indexed: 04/14/2024]
Abstract
Simultaneous Localization and Mapping (SLAM) poses distinct challenges, especially in settings with variable elements, which demand the integration of multiple sensors to ensure robustness. This study addresses these issues by integrating advanced technologies like LiDAR-inertial odometry (LIO), visual-inertial odometry (VIO), and sophisticated Inertial Measurement Unit (IMU) preintegration methods. These integrations enhance the robustness and reliability of the SLAM process for precise mapping of complex environments. Additionally, incorporating an object-detection network aids in identifying and excluding transient objects such as pedestrians and vehicles, essential for maintaining the integrity and accuracy of environmental mapping. The object-detection network features a lightweight design and swift performance, enabling real-time analysis without significant resource utilization. Our approach focuses on harmoniously blending these techniques to yield superior mapping outcomes in complex scenarios. The effectiveness of our proposed methods is substantiated through experimental evaluation, demonstrating their capability to produce more reliable and precise maps in environments with variable elements. The results indicate improvements in autonomous navigation and mapping, providing a practical solution for SLAM in challenging and dynamic settings.
Collapse
Affiliation(s)
- Yiyi Cai
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China;
- The Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China;
- School of Computer and Electronic Information, Guangxi University, Nanning 530000, China
| | - Yang Ou
- The Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China;
- School of Computer and Electronic Information, Guangxi University, Nanning 530000, China
| | - Tuanfa Qin
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China;
- The Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China;
- School of Computer and Electronic Information, Guangxi University, Nanning 530000, China
| |
Collapse
|
10
|
Neyestani A, Picariello F, Ahmed I, Daponte P, De Vito L. From Pixels to Precision: A Survey of Monocular Visual Odometry in Digital Twin Applications. SENSORS (BASEL, SWITZERLAND) 2024; 24:1274. [PMID: 38400432 PMCID: PMC10891866 DOI: 10.3390/s24041274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 02/13/2024] [Accepted: 02/13/2024] [Indexed: 02/25/2024]
Abstract
This survey provides a comprehensive overview of traditional techniques and deep learning-based methodologies for monocular visual odometry (VO), with a focus on displacement measurement applications. This paper outlines the fundamental concepts and general procedures for VO implementation, including feature detection, tracking, motion estimation, triangulation, and trajectory estimation. This paper also explores the research challenges inherent in VO implementation, including scale estimation and ground plane considerations. The scientific literature is rife with diverse methodologies aiming to overcome these challenges, particularly focusing on the problem of accurate scale estimation. This issue has been typically addressed through the reliance on knowledge regarding the height of the camera from the ground plane and the evaluation of feature movements on that plane. Alternatively, some approaches have utilized additional tools, such as LiDAR or depth sensors. This survey of approaches concludes with a discussion of future research challenges and opportunities in the field of monocular visual odometry.
Collapse
Affiliation(s)
| | | | | | | | - Luca De Vito
- Department of Engineering, University of Sannio, 82100 Benevento, Italy; (A.N.); (F.P.); (I.A.); (P.D.)
| |
Collapse
|
11
|
Aldibaja M, Yanase R, Suganuma N. Waypoint Transfer Module between Autonomous Driving Maps Based on LiDAR Directional Sub-Images. SENSORS (BASEL, SWITZERLAND) 2024; 24:875. [PMID: 38339592 PMCID: PMC10857431 DOI: 10.3390/s24030875] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/25/2023] [Revised: 01/15/2024] [Accepted: 01/24/2024] [Indexed: 02/12/2024]
Abstract
Lane graphs are very important for describing road semantics and enabling safe autonomous maneuvers using the localization and path-planning modules. These graphs are considered long-life details because of the rare changes occurring in road structures. On the other hand, the global position of the corresponding topological maps might be changed due to the necessity of updating or extending the maps using different positioning systems such as GNSS/INS-RTK (GIR), Dead-Reckoning (DR), or SLAM technologies. Therefore, the lane graphs should be transferred between maps accurately to describe the same semantics of lanes and landmarks. This paper proposes a unique transfer framework in the image domain based on the LiDAR intensity road surfaces, considering the challenging requirements of its implementation in critical road structures. The road surfaces in a target map are decomposed into directional sub-images with X, Y, and Yaw IDs in the global coordinate system. The XY IDs are used to detect the common areas with a reference map, whereas the Yaw IDs are utilized to reconstruct the vehicle trajectory in the reference map and determine the associated lane graphs. The directional sub-images are then matched to the reference sub-images, and the graphs are safely transferred accordingly. The experimental results have verified the robustness and reliability of the proposed framework to transfer lane graphs safely and accurately between maps, regardless of the complexity of road structures, driving scenarios, map generation methods, and map global accuracies.
Collapse
Affiliation(s)
- Mohammad Aldibaja
- The Advanced Mobility Research Institute, Kanazawa University, Kanazawa 920-1192, Japan; (R.Y.); (N.S.)
| | | | | |
Collapse
|
12
|
Dou H, Wang Z, Wang C, Zhao X. Immediate Pose Recovery Method for Untracked Frames in Feature-Based SLAM. SENSORS (BASEL, SWITZERLAND) 2024; 24:835. [PMID: 38339551 PMCID: PMC10857547 DOI: 10.3390/s24030835] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Revised: 01/22/2024] [Accepted: 01/25/2024] [Indexed: 02/12/2024]
Abstract
In challenging environments, feature-based visual SLAM encounters frequent failures in frame tracking, introducing unknown poses to robotic applications. This paper introduces an immediate approach for recovering untracked camera poses. Through the retrieval of key information from elapsed untracked frames, lost poses are efficiently restored with a short time consumption. Taking account of reconstructed poses and map points during local optimizing, a denser local map is constructed around ambiguous frames to enhance the further SLAM procedure. The proposed method is implemented in a SLAM system, and monocular experiments are conducted on datasets. The experimental results demonstrate that our method can reconstruct the untracked frames in nearly real time, effectively complementing missing segments of the trajectory. Concurrently, the accuracy and robustness for subsequent tracking are improved through the integration of recovered poses and map points.
Collapse
Affiliation(s)
| | | | - Changhong Wang
- Space Control and Inertial Technology Research Center, School of Astronautics, Harbin Institute of Technology, Harbin 150001, China
| | | |
Collapse
|
13
|
Moon S, Lee M. Analyzing the Impact of Objects in an Image on Location Estimation Accuracy in Visual Localization. SENSORS (BASEL, SWITZERLAND) 2024; 24:816. [PMID: 38339532 PMCID: PMC10857014 DOI: 10.3390/s24030816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 01/22/2024] [Accepted: 01/24/2024] [Indexed: 02/12/2024]
Abstract
Visual localization refers to the process of determining an observer's pose by analyzing the spatial relationships between a query image and a pre-existing set of images. In this procedure, matched visual features between images are identified and utilized for pose estimation; consequently, the accuracy of the estimation heavily relies on the precision of feature matching. Incorrect feature matchings, such as those between different objects and/or different points within an object in an image, should thus be avoided. In this paper, our initial evaluation focused on gauging the reliability of each object class within image datasets concerning pose estimation accuracy. This assessment revealed the building class to be reliable, while humans exhibited unreliability across diverse locations. The subsequent study delved deeper into the degradation of pose estimation accuracy by artificially increasing the proportion of the unreliable object-humans. The findings revealed a noteworthy decline started when the average proportion of the humans in the images exceeded 20%. We discuss the results and implications for dataset construction for visual localization.
Collapse
Affiliation(s)
- Sungho Moon
- Department of Information Convergence Engineering, Pusan National University, Busan 46241, Republic of Korea;
| | - Myungho Lee
- School of Computer Science and Engineering, Pusan National University, Busan 46241, Republic of Korea
| |
Collapse
|
14
|
Peng H, Zhao Z, Wang L. A Review of Dynamic Object Filtering in SLAM Based on 3D LiDAR. SENSORS (BASEL, SWITZERLAND) 2024; 24:645. [PMID: 38276337 PMCID: PMC10821332 DOI: 10.3390/s24020645] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 01/17/2024] [Accepted: 01/17/2024] [Indexed: 01/27/2024]
Abstract
SLAM (Simultaneous Localization and Mapping) based on 3D LiDAR (Laser Detection and Ranging) is an expanding field of research with numerous applications in the areas of autonomous driving, mobile robotics, and UAVs (Unmanned Aerial Vehicles). However, in most real-world scenarios, dynamic objects can negatively impact the accuracy and robustness of SLAM. In recent years, the challenge of achieving optimal SLAM performance in dynamic environments has led to the emergence of various research efforts, but there has been relatively little relevant review. This work delves into the development process and current state of SLAM based on 3D LiDAR in dynamic environments. After analyzing the necessity and importance of filtering dynamic objects in SLAM, this paper is developed from two dimensions. At the solution-oriented level, mainstream methods of filtering dynamic targets in 3D point cloud are introduced in detail, such as the ray-tracing-based approach, the visibility-based approach, the segmentation-based approach, and others. Then, at the problem-oriented level, this paper classifies dynamic objects and summarizes the corresponding processing strategies for different categories in the SLAM framework, such as online real-time filtering, post-processing after the mapping, and Long-term SLAM. Finally, the development trends and research directions of dynamic object filtering in SLAM based on 3D LiDAR are discussed and predicted.
Collapse
Affiliation(s)
- Hongrui Peng
- School of Resources and Safety Engineering, Central South University, Changsha 410083, China; (H.P.); (Z.Z.)
| | - Ziyu Zhao
- School of Resources and Safety Engineering, Central South University, Changsha 410083, China; (H.P.); (Z.Z.)
| | - Liguan Wang
- School of Resources and Safety Engineering, Central South University, Changsha 410083, China; (H.P.); (Z.Z.)
- Changsha Digital Mine Co., Ltd., Changsha 410221, China
| |
Collapse
|
15
|
Zhu Y, An H, Wang H, Xu R, Wu M, Lu K. RC-SLAM: Road Constrained Stereo Visual SLAM System Based on Graph Optimization. SENSORS (BASEL, SWITZERLAND) 2024; 24:536. [PMID: 38257631 PMCID: PMC11154416 DOI: 10.3390/s24020536] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Revised: 12/18/2023] [Accepted: 01/12/2024] [Indexed: 01/24/2024]
Abstract
Intelligent vehicles are constrained by road, resulting in a disparity between the assumed six degrees of freedom (DoF) motion within the Visual Simultaneous Localization and Mapping (SLAM) system and the approximate planar motion of vehicles in local areas, inevitably causing additional pose estimation errors. To address this problem, a stereo Visual SLAM system with road constraints based on graph optimization is proposed, called RC-SLAM. Addressing the challenge of representing roads parametrically, a novel method is proposed to approximate local roads as discrete planes and extract parameters of local road planes (LRPs) using homography. Unlike conventional methods, constraints between the vehicle and LRPs are established, effectively mitigating errors arising from assumed six DoF motion in the system. Furthermore, to avoid the impact of depth uncertainty in road features, epipolar constraints are employed to estimate rotation by minimizing the distance between road feature points and epipolar lines, robust rotation estimation is achieved despite depth uncertainties. Notably, a distinctive nonlinear optimization model based on graph optimization is presented, jointly optimizing the poses of vehicle trajectories, LPRs, and map points. The experiments on two datasets demonstrate that the proposed system achieved more accurate estimations of vehicle trajectories by introducing constraints between the vehicle and LRPs. The experiments on a real-world dataset further validate the effectiveness of the proposed system.
Collapse
Affiliation(s)
- Yuan Zhu
- School of Automotive Studies, Tongji University, Shanghai 201800, China; (Y.Z.); (H.A.); (H.W.); (R.X.)
| | - Hao An
- School of Automotive Studies, Tongji University, Shanghai 201800, China; (Y.Z.); (H.A.); (H.W.); (R.X.)
| | - Huaide Wang
- School of Automotive Studies, Tongji University, Shanghai 201800, China; (Y.Z.); (H.A.); (H.W.); (R.X.)
| | - Ruidong Xu
- School of Automotive Studies, Tongji University, Shanghai 201800, China; (Y.Z.); (H.A.); (H.W.); (R.X.)
| | - Mingzhi Wu
- Nanchang Automotive Institute of Intelligence & New Energy, Tongji University, Nanchang 330038, China;
| | - Ke Lu
- School of Automotive Studies, Tongji University, Shanghai 201800, China; (Y.Z.); (H.A.); (H.W.); (R.X.)
| |
Collapse
|
16
|
Gu T, Zhang J, Liu Y. Accurate Monocular SLAM Initialization via Structural Line Tracking. SENSORS (BASEL, SWITZERLAND) 2023; 23:9870. [PMID: 38139717 PMCID: PMC10747762 DOI: 10.3390/s23249870] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Revised: 12/11/2023] [Accepted: 12/14/2023] [Indexed: 12/24/2023]
Abstract
In this paper, we present a novel monocular simultaneous localization and mapping (SLAM) initialization algorithm that relies on structural features by tracking structural lines. This approach addresses the limitations of the traditional method, which can fail to account for a lack of features or their uneven distribution. Our proposed method utilizes a sliding window approach to guarantee the quality and stability of the initial pose estimation. We incorporate multiple geometric constraints, orthogonal dominant directions, and coplanar structural lines to construct an efficient pose optimization strategy. Experimental evaluations conducted on both the collected chessboard datasets and real scene datasets show that our approach provides superior results in terms of accuracy and real-time performance compared to the well-tuned baseline methods. Notably, our algorithm achieves these improvements while being computationally lightweight, without the need for matrix decomposition.
Collapse
Affiliation(s)
- Tianlun Gu
- College of Computer Science, Sichuan University, Chengdu 610065, China; (J.Z.); (Y.L.)
- National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610064, China
| | - Jianwei Zhang
- College of Computer Science, Sichuan University, Chengdu 610065, China; (J.Z.); (Y.L.)
- National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610064, China
| | - Yanli Liu
- College of Computer Science, Sichuan University, Chengdu 610065, China; (J.Z.); (Y.L.)
- National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610064, China
| |
Collapse
|
17
|
Tang Y, Zhao C, Wang J, Zhang C, Sun Q, Zheng WX, Du W, Qian F, Kurths J. Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:9604-9624. [PMID: 35482692 DOI: 10.1109/tnnls.2022.3167688] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Autonomous systems possess the features of inferring their own state, understanding their surroundings, and performing autonomous navigation. With the applications of learning systems, like deep learning and reinforcement learning, the visual-based self-state estimation, environment perception, and navigation capabilities of autonomous systems have been efficiently addressed, and many new learning-based algorithms have surfaced with respect to autonomous visual perception and navigation. In this review, we focus on the applications of learning-based monocular approaches in ego-motion perception, environment perception, and navigation in autonomous systems, which is different from previous reviews that discussed traditional methods. First, we delineate the shortcomings of existing classical visual simultaneous localization and mapping (vSLAM) solutions, which demonstrate the necessity to integrate deep learning techniques. Second, we review the visual-based environmental perception and understanding methods based on deep learning, including deep learning-based monocular depth estimation, monocular ego-motion prediction, image enhancement, object detection, semantic segmentation, and their combinations with traditional vSLAM frameworks. Then, we focus on the visual navigation based on learning systems, mainly including reinforcement learning and deep reinforcement learning. Finally, we examine several challenges and promising directions discussed and concluded in related research of learning systems in the era of computer science and robotics.
Collapse
|
18
|
Liu H, Xue H, Zhao L, Chen D, Peng Z, Zhang G. MagLoc-AR: Magnetic-Based Localization for Visual-Free Augmented Reality in Large-Scale Indoor Environments. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2023; 29:4383-4393. [PMID: 37782616 DOI: 10.1109/tvcg.2023.3321088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/04/2023]
Abstract
Accurate localization of a display device is essential for AR in large-scale environments. Visual-based localization is the most commonly used solution, but poses privacy risks, suffers from robustness issues and consumes high power. Wireless signal-based localization is a potential visual-free solution, but its accuracy is not enough for AR. In this paper, we present MagLoc-AR, a novel visual-free localization solution that achieves sufficient accuracy for some AR applications (e.g. AR navigation) in large-scale indoor environments. We exploit the location-dependent magnetic field interference that is ubiquitous indoors as a localization signal. Our method requires only a consumer-grade 9-axis IMU, with the gyroscope and acceleration measurements used to recover the motion trajectory, and the magnetic measurements used to register the trajectory to the global map. To meet the accuracy requirement of AR, we propose a mapping method to reconstruct a globally consistent magnetic field of the environment, and a localization method fusing the biased magnetic measurements with the network-predicted motion to improve localization accuracy. In addition, we provide the first dataset for both visual-based and geomagnetic-based localization in large-scale indoor environments. Evaluations on the dataset demonstrate that our proposed method is sufficiently accurate for AR navigation and has advantages over the visual-based methods in terms of power consumption and robustness. Project page: https://github.com/zju3dv/MagLoc-AR/.
Collapse
|
19
|
Fontana E, Lodi Rizzini D. Accurate Global Point Cloud Registration Using GPU-Based Parallel Angular Radon Spectrum. SENSORS (BASEL, SWITZERLAND) 2023; 23:8628. [PMID: 37896722 PMCID: PMC10611382 DOI: 10.3390/s23208628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 10/16/2023] [Accepted: 10/17/2023] [Indexed: 10/29/2023]
Abstract
Accurate robot localization and mapping can be improved through the adoption of globally optimal registration methods, like the Angular Radon Spectrum (ARS). In this paper, we present Cud-ARS, an efficient variant of the ARS algorithm for 2D registration designed for parallel execution of the most computationally expensive steps on Nvidia™ Graphics Processing Units (GPUs). Cud-ARS is able to compute the ARS in parallel blocks, with each associated to a subset of input points. We also propose a global branch-and-bound method for translation estimation. This novel parallel algorithm has been tested on multiple datasets. The proposed method is able to speed up the execution time by two orders of magnitude while obtaining more accurate results in rotation estimation than state-of-the-art correspondence-based algorithms. Our experiments also assess the potential of this novel approach in mapping applications, showing the contribution of GPU programming to efficient solutions of robotic tasks.
Collapse
Affiliation(s)
- Ernesto Fontana
- Department of Engineering and Architecture, University of Parma, Parco Area delle Scienze 181/A, 43124 Parma, Italy;
| | - Dario Lodi Rizzini
- Department of Engineering and Architecture, University of Parma, Parco Area delle Scienze 181/A, 43124 Parma, Italy;
- Interdepartmental Center for Energy and Environment (CIDEA), University of Parma, Parco Area delle Scienze 95, 43124 Parma, Italy
| |
Collapse
|
20
|
Wang Y, Zhang Y, Hu L, Wang W, Ge G, Tan S. A Semantic Topology Graph to Detect Re-Localization and Loop Closure of the Visual Simultaneous Localization and Mapping System in a Dynamic Environment. SENSORS (BASEL, SWITZERLAND) 2023; 23:8445. [PMID: 37896538 PMCID: PMC10611121 DOI: 10.3390/s23208445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 10/02/2023] [Accepted: 10/10/2023] [Indexed: 10/29/2023]
Abstract
Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.
Collapse
Affiliation(s)
- Yang Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Yi Zhang
- Advanced Manufacturing and Automatization Engineering Laboratory, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Lihe Hu
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Wei Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Gengyu Ge
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| | - Shuyi Tan
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; (Y.W.); (L.H.); (W.W.); (G.G.); (S.T.)
| |
Collapse
|
21
|
Zhang XY, Abd Rahman AH, Qamar F. Semantic visual simultaneous localization and mapping (SLAM) using deep learning for dynamic scenes. PeerJ Comput Sci 2023; 9:e1628. [PMID: 37869467 PMCID: PMC10588701 DOI: 10.7717/peerj-cs.1628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 09/11/2023] [Indexed: 10/24/2023]
Abstract
Simultaneous localization and mapping (SLAM) is a fundamental problem in robotics and computer vision. It involves the task of a robot or an autonomous system navigating an unknown environment, simultaneously creating a map of the surroundings, and accurately estimating its position within that map. While significant progress has been made in SLAM over the years, challenges still need to be addressed. One prominent issue is robustness and accuracy in dynamic environments, which can cause uncertainties and errors in the estimation process. Traditional methods using temporal information to differentiate static and dynamic objects have limitations in accuracy and applicability. Nowadays, many research trends have leaned towards utilizing deep learning-based methods which leverage the capabilities to handle dynamic objects, semantic segmentation, and motion estimation, aiming to improve accuracy and adaptability in complex scenes. This article proposed an approach to enhance monocular visual odometry's robustness and precision in dynamic environments. An enhanced algorithm using the semantic segmentation algorithm DeeplabV3+ is used to identify dynamic objects in the image and then apply the motion consistency check to remove feature points belonging to dynamic objects. The remaining static feature points are then used for feature matching and pose estimation based on ORB-SLAM2 using the Technical University of Munich (TUM) dataset. Experimental results show that our method outperforms traditional visual odometry methods in accuracy and robustness, especially in dynamic environments. By eliminating the influence of moving objects, our method improves the accuracy and robustness of visual odometry in dynamic environments. Compared to the traditional ORB-SLAM2, the results show that the system significantly reduces the absolute trajectory error and the relative pose error in dynamic scenes. Our approach has significantly improved the accuracy and robustness of the SLAM system's pose estimation.
Collapse
Affiliation(s)
- Xiao Ya Zhang
- Center for Artificial Intelligence Technology, Universiti Kebangsaan Malaysia, Bangi, Malaysia
| | - Abdul Hadi Abd Rahman
- Center for Artificial Intelligence Technology, Universiti Kebangsaan Malaysia, Bangi, Malaysia
| | - Faizan Qamar
- Center for Cyber Security, Universiti Kebangsaan Malaysia, Bangi, Malaysia
| |
Collapse
|
22
|
Abd Rahman NA, Mohamed Sahari KS, Baharuddin MZ. The coverage and detection limit of a sampling point for robotics radiation mapping. Appl Radiat Isot 2023; 200:110968. [PMID: 37544032 DOI: 10.1016/j.apradiso.2023.110968] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Revised: 07/26/2023] [Accepted: 07/30/2023] [Indexed: 08/08/2023]
Abstract
The sensor coverage problem aims to maximize the coverage of a target area with a fixed or minimum number of sensors. However, the sampling point coverage for radiation mapping has yet to be specified or adequately established. When dealing with unknown radiation fields, it is critical that the placements of sampling points will ensure that all hotspots are detected and accurately identified. Therefore, the concept of coverage and detection limit for a sampling point in radiation mapping is proposed in this paper. The proposed concept relates the angular dependency of the radiation measurement instruments with the detector detection limit or minimum detectable amount (MDA). To demonstrate the implementation, the concept is used to compute the sensitivity of the radiation map for coverage radiation mapping with mobile robot. Simulation results showed that hotspots with intensity equal to or above the sampling point detection limit were successfully detected regardless of their position within the coverage circle. Moreover, the experimental results of coverage radiation mapping showed that the concept can be used to compute the resolution of the radiation map. This will help the user to efficiently configure the appropriate grid size that suit their mapping situation and requirements.
Collapse
Affiliation(s)
- Nur Aira Abd Rahman
- Malaysian Nuclear Agency, 43000, Bangi, Selangor, Malaysia; Universiti Tenaga Nasional (UNITEN), 43000, Kajang, Selangor, Malaysia.
| | | | | |
Collapse
|
23
|
Ahmed MF, Masood K, Fremont V, Fantoni I. Active SLAM: A Review on Last Decade. SENSORS (BASEL, SWITZERLAND) 2023; 23:8097. [PMID: 37836928 PMCID: PMC10575033 DOI: 10.3390/s23198097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 09/18/2023] [Accepted: 09/21/2023] [Indexed: 10/15/2023]
Abstract
This article presents a comprehensive review of the Active Simultaneous Localization and Mapping (A-SLAM) research conducted over the past decade. It explores the formulation, applications, and methodologies employed in A-SLAM, particularly in trajectory generation and control-action selection, drawing on concepts from Information Theory (IT) and the Theory of Optimal Experimental Design (TOED). This review includes both qualitative and quantitative analyses of various approaches, deployment scenarios, configurations, path-planning methods, and utility functions within A-SLAM research. Furthermore, this article introduces a novel analysis of Active Collaborative SLAM (AC-SLAM), focusing on collaborative aspects within SLAM systems. It includes a thorough examination of collaborative parameters and approaches, supported by both qualitative and statistical assessments. This study also identifies limitations in the existing literature and suggests potential avenues for future research. This survey serves as a valuable resource for researchers seeking insights into A-SLAM methods and techniques, offering a current overview of A-SLAM formulation.
Collapse
Affiliation(s)
- Muhammad Farhan Ahmed
- Laboratoire des Sciences du Numérique de Nantes (LS2N), CNRS, Ecole Centrale de Nantes, 1 Rue de la Noë, 44300 Nantes, France; (M.F.A.); (I.F.)
| | - Khayyam Masood
- Capgemini Engineering, 4 Avenue Didier Daurat, 31700 Blagnac, France;
| | - Vincent Fremont
- Laboratoire des Sciences du Numérique de Nantes (LS2N), CNRS, Ecole Centrale de Nantes, 1 Rue de la Noë, 44300 Nantes, France; (M.F.A.); (I.F.)
| | - Isabelle Fantoni
- Laboratoire des Sciences du Numérique de Nantes (LS2N), CNRS, Ecole Centrale de Nantes, 1 Rue de la Noë, 44300 Nantes, France; (M.F.A.); (I.F.)
| |
Collapse
|
24
|
Pham MD, D’Angiulli A, Dehnavi MM, Chhabra R. From Brain Models to Robotic Embodied Cognition: How Does Biological Plausibility Inform Neuromorphic Systems? Brain Sci 2023; 13:1316. [PMID: 37759917 PMCID: PMC10526461 DOI: 10.3390/brainsci13091316] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Revised: 09/05/2023] [Accepted: 09/07/2023] [Indexed: 09/29/2023] Open
Abstract
We examine the challenging "marriage" between computational efficiency and biological plausibility-A crucial node in the domain of spiking neural networks at the intersection of neuroscience, artificial intelligence, and robotics. Through a transdisciplinary review, we retrace the historical and most recent constraining influences that these parallel fields have exerted on descriptive analysis of the brain, construction of predictive brain models, and ultimately, the embodiment of neural networks in an enacted robotic agent. We study models of Spiking Neural Networks (SNN) as the central means enabling autonomous and intelligent behaviors in biological systems. We then provide a critical comparison of the available hardware and software to emulate SNNs for investigating biological entities and their application on artificial systems. Neuromorphics is identified as a promising tool to embody SNNs in real physical systems and different neuromorphic chips are compared. The concepts required for describing SNNs are dissected and contextualized in the new no man's land between cognitive neuroscience and artificial intelligence. Although there are recent reviews on the application of neuromorphic computing in various modules of the guidance, navigation, and control of robotic systems, the focus of this paper is more on closing the cognition loop in SNN-embodied robotics. We argue that biologically viable spiking neuronal models used for electroencephalogram signals are excellent candidates for furthering our knowledge of the explainability of SNNs. We complete our survey by reviewing different robotic modules that can benefit from neuromorphic hardware, e.g., perception (with a focus on vision), localization, and cognition. We conclude that the tradeoff between symbolic computational power and biological plausibility of hardware can be best addressed by neuromorphics, whose presence in neurorobotics provides an accountable empirical testbench for investigating synthetic and natural embodied cognition. We argue this is where both theoretical and empirical future work should converge in multidisciplinary efforts involving neuroscience, artificial intelligence, and robotics.
Collapse
Affiliation(s)
- Martin Do Pham
- Department of Computer Science, University of Toronto, Toronto, ON M5S 1A1, Canada; (M.D.P.); (M.M.D.)
| | - Amedeo D’Angiulli
- Department of Neuroscience, Carleton University, Ottawa, ON K1S 5B6, Canada;
| | - Maryam Mehri Dehnavi
- Department of Computer Science, University of Toronto, Toronto, ON M5S 1A1, Canada; (M.D.P.); (M.M.D.)
| | - Robin Chhabra
- Department of Mechanical and Aerospace Engineering, Carleton University, Ottawa, ON K1S 5B6, Canada
| |
Collapse
|
25
|
Wu J, Ren H, Lin T, Yao Y, Fang Z, Liu C. A Pure Electric Driverless Crawler Construction Machinery Walking Method Based on the Fusion SLAM and Improved Pure Pursuit Algorithms. SENSORS (BASEL, SWITZERLAND) 2023; 23:7784. [PMID: 37765841 PMCID: PMC10537430 DOI: 10.3390/s23187784] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Revised: 09/07/2023] [Accepted: 09/08/2023] [Indexed: 09/29/2023]
Abstract
Driverless technology refers to the technology that vehicles use to drive independently with the help of driverless system under the condition of unmanned intervention. The working environment of construction machinery is bad, and the working conditions are complex. The use of driverless technology can greatly reduce the risk of driver operation, reduce labor costs and improve economic benefits.Aiming at the problem of the GPS positioning signal in the working environment of construction machinery being weak and not able to achieve accurate positioning, this paper uses the fusion SLAM algorithm based on improved NDT to realize the real-time positioning of the whole vehicle through reconstruction of the scene. Considering that the motion characteristics of crawler construction machinery are different from those of ordinary passenger cars, this paper improves the existing pure pursuit algorithm. Simulations and real vehicle tests show that the algorithm combined with the fusion SLAM algorithm can realize the motion control of driverless crawler construction machinery well, complete the tracking of the set trajectory perfectly and have high robustness. Considering that there is no mature walking method of driverless crawler construction machinery for reference, the research of this paper will lay a foundation for the development of driverless crawler construction machinery.
Collapse
Affiliation(s)
- Jiangdong Wu
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
- Fujian Key Laboratory of Green Intelligent Drive and Transmission for Mobile Machinery, Xiamen 361021, China
| | - Haoling Ren
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
- Fujian Key Laboratory of Green Intelligent Drive and Transmission for Mobile Machinery, Xiamen 361021, China
| | - Tianliang Lin
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
- Fujian Key Laboratory of Green Intelligent Drive and Transmission for Mobile Machinery, Xiamen 361021, China
| | - Yu Yao
- Mechatronic Engineering with the School of Beihang University, Beijing 102206, China
| | - Zhen Fang
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
- Fujian Key Laboratory of Green Intelligent Drive and Transmission for Mobile Machinery, Xiamen 361021, China
| | - Chang Liu
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
- Fujian Key Laboratory of Green Intelligent Drive and Transmission for Mobile Machinery, Xiamen 361021, China
| |
Collapse
|
26
|
Gao R, Wan Z, Guo S, Jiang C, Zhang Y. LFVB-BioSLAM: A Bionic SLAM System with a Light-Weight LiDAR Front End and a Bio-Inspired Visual Back End. Biomimetics (Basel) 2023; 8:410. [PMID: 37754161 PMCID: PMC10526866 DOI: 10.3390/biomimetics8050410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2023] [Revised: 08/17/2023] [Accepted: 09/01/2023] [Indexed: 09/28/2023] Open
Abstract
Simultaneous localization and mapping (SLAM) is one of the crucial techniques applied in autonomous robot navigation. The majority of present popular SLAM algorithms are built within probabilistic optimization frameworks, achieving high accuracy performance at the expense of high power consumption and latency. In contrast to robots, animals are born with the capability to efficiently and robustly navigate in nature, and bionic SLAM algorithms have received increasing attention recently. Current bionic SLAM algorithms, including RatSLAM, with relatively low accuracy and robustness, tend to fail in certain challenging environments. In order to design a bionic SLAM system with a novel framework and relatively high practicality, and to facilitate the development of bionic SLAM research, in this paper we present LFVB-BioSLAM, a bionic SLAM system with a light-weight LiDAR-based front end and a bio-inspired vision-based back end. We adopt a range flow-based LiDAR odometry as the front end of the SLAM system, providing the odometry estimation for the back end, and we propose a biologically-inspired back end processing algorithm based on the monocular RGB camera, performing loop closure detection and path integration. Our method is verified through real-world experiments, and the results show that LFVB-BioSLAM outperforms RatSLAM, a vision-based bionic SLAM algorithm, and RF2O, a laser-based horizontal planar odometry algorithm, in terms of accuracy and robustness.
Collapse
Affiliation(s)
- Ruilan Gao
- State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China; (R.G.); (Z.W.); (S.G.); (C.J.)
| | - Zeyu Wan
- State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China; (R.G.); (Z.W.); (S.G.); (C.J.)
| | - Sitong Guo
- State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China; (R.G.); (Z.W.); (S.G.); (C.J.)
| | - Changjian Jiang
- State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China; (R.G.); (Z.W.); (S.G.); (C.J.)
| | - Yu Zhang
- State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China; (R.G.); (Z.W.); (S.G.); (C.J.)
- Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province, Hangzhou 310027, China
| |
Collapse
|
27
|
Wang S, Su D, Li M, Jiang Y, Zhang L, Yan H, Hu N, Tan Y. LFSD: a VSLAM dataset with plant detection and tracking in lettuce farm. FRONTIERS IN PLANT SCIENCE 2023; 14:1175743. [PMID: 37705704 PMCID: PMC10497103 DOI: 10.3389/fpls.2023.1175743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 08/01/2023] [Indexed: 09/15/2023]
Affiliation(s)
- Shuo Wang
- College of Engineering, China Agricultural University, Beijing, China
| | - Daobilige Su
- College of Engineering, China Agricultural University, Beijing, China
| | - Maofeng Li
- Beijing Zhong Nong LV Tong Agriculture Development LTD, Beijing, China
| | - Yiyu Jiang
- College of Engineering, China Agricultural University, Beijing, China
| | - Lina Zhang
- College of Engineering, China Agricultural University, Beijing, China
| | - Hao Yan
- College of Engineering, China Agricultural University, Beijing, China
| | - Nan Hu
- College of Engineering, China Agricultural University, Beijing, China
| | - Yu Tan
- College of Engineering, China Agricultural University, Beijing, China
| |
Collapse
|
28
|
Yu X, Zhao J, Wu H, Wang A. A Novel Evaluation Method for SLAM-Based 3D Reconstruction of Lumen Panoramas. SENSORS (BASEL, SWITZERLAND) 2023; 23:7188. [PMID: 37631725 PMCID: PMC10459170 DOI: 10.3390/s23167188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 08/09/2023] [Accepted: 08/10/2023] [Indexed: 08/27/2023]
Abstract
Laparoscopy is employed in conventional minimally invasive surgery to inspect internal cavities by viewing two-dimensional images on a monitor. This method has a limited field of view and provides insufficient information for surgeons, increasing surgical complexity. Utilizing simultaneous localization and mapping (SLAM) technology to reconstruct laparoscopic scenes can offer more comprehensive and intuitive visual feedback. Moreover, the precision of the reconstructed models is a crucial factor for further applications of surgical assistance systems. However, challenges such as data scarcity and scale uncertainty hinder effective assessment of the accuracy of endoscopic monocular SLAM reconstructions. Therefore, this paper proposes a technique that incorporates existing knowledge from calibration objects to supplement metric information and resolve scale ambiguity issues, and it quantifies the endoscopic reconstruction accuracy based on local alignment metrics. The experimental results demonstrate that the reconstructed models restore realistic scales and enable error analysis for laparoscopic SLAM reconstruction systems. This suggests that for the evaluation of monocular SLAM three-dimensional (3D) reconstruction accuracy in minimally invasive surgery scenarios, our proposed scheme for recovering scale factors is viable, and our evaluation outcomes can serve as criteria for measuring reconstruction precision.
Collapse
Affiliation(s)
- Xiaoyu Yu
- College of Electron and Information, University of Electronic Science and Technology of China, Zhongshan Institute, Zhongshan 528402, China;
- Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China (A.W.)
| | - Jianbo Zhao
- Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China (A.W.)
| | - Haibin Wu
- Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China (A.W.)
| | - Aili Wang
- Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China (A.W.)
| |
Collapse
|
29
|
Maleki B, Falque R, Vidal-Calleja T, Alempijevic A. SPaM: soft patch matching for non-rigid pointcloud registration. Front Robot AI 2023; 10:1019579. [PMID: 37529483 PMCID: PMC10387535 DOI: 10.3389/frobt.2023.1019579] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2022] [Accepted: 05/09/2023] [Indexed: 08/03/2023] Open
Abstract
3d reconstruction of deformable objects in dynamic scenes forms the fundamental basis of many robotic applications. Existing mesh-based approaches compromise registration accuracy, and lose important details due to interpolation and smoothing. Additionally, existing non-rigid registration techniques struggle with unindexed points and disconnected manifolds. We propose a novel non-rigid registration framework for raw, unstructured, deformable point clouds purely based on geometric features. The global non-rigid deformation of an object is formulated as an aggregation of locally rigid transformations. The concept of locality is embodied in soft patches described by geometrical properties based on SHOT descriptor and its neighborhood. By considering the confidence score of pairwise association between soft patches of two scans (not necessarily consecutive), a computed similarity matrix serves as the seed to grow a correspondence graph which leverages rigidity terms defined in As-Rigid-As-Possible for pruning and optimization. Experiments on simulated and publicly available datasets demonstrate the capability of the proposed approach to cope with large deformations blended with numerous missing parts in the scan process.
Collapse
|
30
|
Liang L, Pei H. Affine Iterative Closest Point Algorithm Based on Color Information and Correntropy for Precise Point Set Registration. SENSORS (BASEL, SWITZERLAND) 2023; 23:6475. [PMID: 37514769 PMCID: PMC10383488 DOI: 10.3390/s23146475] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/21/2023] [Revised: 06/26/2023] [Accepted: 06/30/2023] [Indexed: 07/30/2023]
Abstract
In this paper, we propose a novel affine iterative closest point algorithm based on color information and correntropy, which can effectively deal with the registration problems with a large number of noise and outliers and small deformations in RGB-D datasets. Firstly, to alleviate the problem of low registration accuracy for data with weak geometric structures, we consider introducing color features into traditional affine algorithms to establish more accurate and reliable correspondences. Secondly, we introduce the correntropy measurement to overcome the influence of a large amount of noise and outliers in the RGB-D datasets, thereby further improving the registration accuracy. Experimental results demonstrate that the proposed registration algorithm has higher registration accuracy, with error reduction of almost 10 times, and achieves more stable robustness than other advanced algorithms.
Collapse
Affiliation(s)
- Lexian Liang
- Key Laboratory of Autonomous Systems and Networked Control, Ministry of Education, Unmanned Aerial Vehicle Systems Engineering Technology Research Center of Guangdong, South China University of Technology, Guangzhou 510640, China
| | - Hailong Pei
- Key Laboratory of Autonomous Systems and Networked Control, Ministry of Education, Unmanned Aerial Vehicle Systems Engineering Technology Research Center of Guangdong, South China University of Technology, Guangzhou 510640, China
| |
Collapse
|
31
|
Zha B, Yilmaz A. Subgraph Learning for Topological Geolocalization with Graph Neural Networks. SENSORS (BASEL, SWITZERLAND) 2023; 23:5098. [PMID: 37299825 PMCID: PMC10255631 DOI: 10.3390/s23115098] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/29/2023] [Revised: 05/24/2023] [Accepted: 05/24/2023] [Indexed: 06/12/2023]
Abstract
One of the challenges of spatial cognition, such as self-localization and navigation, is to develop an efficient learning approach capable of mimicking human ability. This paper proposes a novel approach for topological geolocalization on the map using motion trajectory and graph neural networks. Specifically, our learning method learns an embedding of the motion trajectory encoded as a path subgraph where the node and edge represent turning direction and relative distance information by training a graph neural network. We formulate the subgraph learning as a multi-class classification problem in which the output node IDs are interpreted as the object's location on the map. After training using three map datasets with small, medium, and large sizes, the node localization tests on simulated trajectories generated from the map show 93.61%, 95.33%, and 87.50% accuracy, respectively. We also demonstrate similar accuracy for our approach on actual trajectories generated by visual-inertial odometry. The key benefits of our approach are as follows: (1) we take advantage of the powerful graph-modeling ability of neural graph networks, (2) it only requires a map in the form of a 2D graph, and (3) it only requires an affordable sensor that generates relative motion trajectory.
Collapse
Affiliation(s)
- Bing Zha
- Photogrammetric Computer Vision Lab, Department of Civil, Environmental and Geodetic Engineering, The Ohio State University, Columbus, OH 43210, USA;
| | | |
Collapse
|
32
|
Bavle H, Sanchez-Lopez JL, Cimarelli C, Tourani A, Voos H. From SLAM to Situational Awareness: Challenges and Survey. SENSORS (BASEL, SWITZERLAND) 2023; 23:4849. [PMID: 37430762 DOI: 10.3390/s23104849] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 04/27/2023] [Accepted: 05/13/2023] [Indexed: 07/12/2023]
Abstract
The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic situational awareness by discussing interesting recent research directions.
Collapse
Affiliation(s)
- Hriday Bavle
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Jose Luis Sanchez-Lopez
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Claudio Cimarelli
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Ali Tourani
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
| | - Holger Voos
- Interdisciplinary Center for Security Reliability and Trust (SnT), University of Luxembourg, 1855 Luxembourg, Luxembourg
- Department of Engineering, Faculty of Science, Technology, and Medicine (FSTM), University of Luxembourg, 1359 Luxembourg, Luxembourg
| |
Collapse
|
33
|
Huang SS, Chen H, Huang J, Fu H, Hu SM. Real-Time Globally Consistent 3D Reconstruction With Semantic Priors. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2023; 29:1977-1991. [PMID: 34941511 DOI: 10.1109/tvcg.2021.3137912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Maintaining global consistency continues to be critical for online 3D indoor scene reconstruction. However, it is still challenging to generate satisfactory 3D reconstruction in terms of global consistency for previous approaches using purely geometric analysis, even with bundle adjustment or loop closure techniques. In this article, we propose a novel real-time 3D reconstruction approach which effectively integrates both semantic and geometric cues. The key challenge is how to map this indicative information, i.e., semantic priors, into a metric space as measurable information, thus enabling more accurate semantic fusion leveraging both the geometric and semantic cues. To this end, we introduce a semantic space with a continuous metric function measuring the distance between discrete semantic observations. Within the semantic space, we present an accurate frame-to-model semantic tracker for camera pose estimation, and semantic pose graph equipped with semantic links between submaps for globally consistent 3D scene reconstruction. With extensive evaluation on public synthetic and real-world 3D indoor scene RGB-D datasets, we show that our approach outperforms the previous approaches for 3D scene reconstruction both quantitatively and qualitatively, especially in terms of global consistency.
Collapse
|
34
|
Roy R, Tu YP, Sheu LJ, Chieng WH, Tang LC, Ismail H. Path Planning and Motion Control of Indoor Mobile Robot under Exploration-Based SLAM (e-SLAM). SENSORS (BASEL, SWITZERLAND) 2023; 23:3606. [PMID: 37050664 PMCID: PMC10099333 DOI: 10.3390/s23073606] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Revised: 03/27/2023] [Accepted: 03/28/2023] [Indexed: 06/19/2023]
Abstract
Indoor mobile robot (IMR) motion control for e-SLAM techniques with limited sensors, i.e., only LiDAR, is proposed in this research. The path was initially generated from simple floor plans constructed by the IMR exploration. The path planning starts from the vertices which can be traveled through, proceeds to the velocity planning on both cornering and linear motion, and reaches the interpolated discrete points joining the vertices. The IMR recognizes its location and environment gradually from the LiDAR data. The study imposes the upper rings of the LiDAR image to perform localization while the lower rings are for obstacle detection. The IMR must travel through a series of featured vertices and perform the path planning further generating an integrated LiDAR image. A considerable challenge is that the LiDAR data are the only source to be compared with the path planned according to the floor map. Certain changes still need to be adapted into, for example, the distance precision with relevance to the floor map and the IMR deviation in order to avoid obstacles on the path. The LiDAR setting and IMR speed regulation account for a critical issue. The study contributed to integrating a step-by-step procedure of implementing path planning and motion control using solely the LiDAR data along with the integration of various pieces of software. The control strategy is thus improved while experimenting with various proportional control gains for position, orientation, and velocity of the LiDAR in the IMR.
Collapse
Affiliation(s)
- Rohit Roy
- Department of Mechanical Engineering, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan; (R.R.); (Y.-P.T.); (L.-C.T.)
| | - You-Peng Tu
- Department of Mechanical Engineering, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan; (R.R.); (Y.-P.T.); (L.-C.T.)
| | - Long-Jye Sheu
- Department of Mechanical Engineering, Chung Hua University, Hsinchu 30012, Taiwan;
| | - Wei-Hua Chieng
- Department of Mechanical Engineering, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan; (R.R.); (Y.-P.T.); (L.-C.T.)
| | - Li-Chuan Tang
- Department of Mechanical Engineering, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan; (R.R.); (Y.-P.T.); (L.-C.T.)
| | - Hasan Ismail
- Jurusen Teknik Mesin, Universitas Negeri Malang, Malang 55165, Indonesia;
| |
Collapse
|
35
|
Frosi M, Gobbi V, Matteucci M. OSM-SLAM: Aiding SLAM with OpenStreetMaps priors. Front Robot AI 2023; 10:1064934. [PMID: 37064577 PMCID: PMC10090495 DOI: 10.3389/frobt.2023.1064934] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2022] [Accepted: 03/15/2023] [Indexed: 03/31/2023] Open
Abstract
In the last decades, Simultaneous Localization and Mapping (SLAM) proved to be a fundamental topic in the field of robotics, due to the many applications, ranging from autonomous driving to 3D reconstruction. Many systems have been proposed in literature exploiting a heterogeneous variety of sensors. State-of-the-art methods build their own map from scratch, using only data coming from the equipment of the robot, and not exploiting possible reconstructions of the environment. Moreover, temporary loss of data proves to be a challenge for SLAM systems, as it demands efficient re-localization to continue the localization process. In this paper, we present a SLAM system that exploits additional information coming from mapping services like OpenStreetMaps, hence the name OSM-SLAM, to face these issues. We extend an existing LiDAR-based Graph SLAM system, ART-SLAM, making it able to integrate the 2D geometry of buildings in the trajectory estimation process, by matching a prior OpenStreetMaps map with a single LiDAR scan. Each estimated pose of the robot is then associated with all buildings surrounding it. This association allows to improve localization accuracy, but also to adjust possible mistakes in the prior map. The pose estimates coming from SLAM are then jointly optimized with the constraints associated with the various OSM buildings, which can assume one of the following types: Buildings are always fixed (Prior SLAM); buildings surrounding a robot are movable in chunks, for every scan (Rigid SLAM); and every single building is free to move independently from the others (Non-rigid SLAM). Lastly, OSM maps can also be used to re-localize the robot when sensor data is lost. We compare the accuracy of the proposed system with existing methods for LiDAR-based SLAM, including the baseline, also providing a visual inspection of the results. The comparison is made by evaluating the estimated trajectory displacement using the KITTI odometry dataset. Moreover, the experimental campaign, along with an ablation study on the re-localization capabilities of the proposed system and its accuracy in loop detection-denied scenarios, allow a discussion about how the quality of prior maps influences the SLAM procedure, which may lead to worse estimates than the baseline.
Collapse
|
36
|
Wang Z, Shen M, Chen Q. Eliminating Scale Ambiguity of Unsupervised Monocular Visual Odometry. Neural Process Lett 2023. [DOI: 10.1007/s11063-023-11224-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/28/2023]
|
37
|
Jin J, Jiang X, Yu C, Zhao L, Tang Z. Dynamic visual simultaneous localization and mapping based on semantic segmentation module. APPL INTELL 2023. [DOI: 10.1007/s10489-023-04531-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/07/2023]
|
38
|
Tao J, Dai W, Kong D, Wan J, He B, Zhang Y. Drift‐free localisation using prior cross‐source map for indoor low‐light environments. IET CYBER-SYSTEMS AND ROBOTICS 2023. [DOI: 10.1049/csy2.12081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023] Open
Affiliation(s)
- Junyi Tao
- State Key Laboratory of Industrial Control Technology College of Control Science and Engineering Zhejiang University Hangzhou China
| | - Weichen Dai
- School of Computer Science Hangzhou Dianzi University Hangzhou China
| | - Da Kong
- State Key Laboratory of Industrial Control Technology College of Control Science and Engineering Zhejiang University Hangzhou China
| | - Jiayan Wan
- State Key Laboratory of Industrial Control Technology College of Control Science and Engineering Zhejiang University Hangzhou China
| | - Bin He
- State Key Laboratory of Industrial Control Technology College of Control Science and Engineering Zhejiang University Hangzhou China
| | - Yu Zhang
- State Key Laboratory of Industrial Control Technology College of Control Science and Engineering Zhejiang University Hangzhou China
- Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province Hangzhou China
| |
Collapse
|
39
|
A Real-Time Monocular Visual SLAM Based on the Bundle Adjustment with Adaptive Robust Kernel. J INTELL ROBOT SYST 2023. [DOI: 10.1007/s10846-023-01817-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2023]
|
40
|
Tchuiev V, Indelman V. Epistemic Uncertainty Aware Semantic Localization and Mapping for Inference and Belief Space Planning. ARTIF INTELL 2023. [DOI: 10.1016/j.artint.2023.103903] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023]
|
41
|
Shimoda T, Koga S, Sato K. Autonomous Motion Control of a Mobile Robot Using Marker Recognition via Deep Learning in GPS-Denied Environments. JOURNAL OF ROBOTICS AND MECHATRONICS 2023. [DOI: 10.20965/jrm.2023.p0136] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/21/2023]
Abstract
In this study, an autonomous traveling control system for a mobile robot was developed using the results for calculation of the relative positions and angles between the mobile robot and markers, based on the image information obtained from a camera mounted on the mobile robot. The mobile robot runs autonomously based on the path of the marker. However, as the conventional method uses OpenCV to identify the shape of the marker using the color information of the marker, the marker may be misrecognized owing to the influence of light. Furthermore, the specifications of the camera limit the detection distance of the marker placed opposite it, which inevitably limits the straight traveling distance of the mobile robot in the proposed method. The proposed method improves the accuracy of marker recognition by using deep learning, and also devises the method of placing markers that allows the user to move straight ahead over a longer distance. It can also easily achieve autonomous path travel control, including long-distance straight lines, for a mobile robot in an environment where global positioning systems (GPS) cannot be received. In addition, the system can be easily operated by an actual user, who need not have any knowledge of programming, because the travel path of the mobile robot can be set up simply by placing markers. The effectiveness of the proposed system was demonstrated through several experiments.
Collapse
Affiliation(s)
- Takashi Shimoda
- Department of Mechanical Engineering, Faculty of Science and Engineering, Saga University, 1 Honjo Saga, Saga 840-8502, Japan
| | - Shoya Koga
- Department of Mechanical Engineering, Faculty of Science and Engineering, Saga University, 1 Honjo Saga, Saga 840-8502, Japan
| | - Kazuya Sato
- Department of Mechanical Engineering, Faculty of Science and Engineering, Saga University, 1 Honjo Saga, Saga 840-8502, Japan
| |
Collapse
|
42
|
LiDAR-Based Sensor Fusion SLAM and Localization for Autonomous Driving Vehicles in Complex Scenarios. J Imaging 2023; 9:jimaging9020052. [PMID: 36826971 PMCID: PMC9961341 DOI: 10.3390/jimaging9020052] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2022] [Revised: 01/21/2023] [Accepted: 02/13/2023] [Indexed: 02/22/2023] Open
Abstract
LiDAR-based simultaneous localization and mapping (SLAM) and online localization methods are widely used in autonomous driving, and are key parts of intelligent vehicles. However, current SLAM algorithms have limitations in map drift and localization algorithms based on a single sensor have poor adaptability to complex scenarios. A SLAM and online localization method based on multi-sensor fusion is proposed and integrated into a general framework in this paper. In the mapping process, constraints consisting of normal distributions transform (NDT) registration, loop closure detection and real time kinematic (RTK) global navigation satellite system (GNSS) position for the front-end and the pose graph optimization algorithm for the back-end, which are applied to achieve an optimized map without drift. In the localization process, the error state Kalman filter (ESKF) fuses LiDAR-based localization position and vehicle states to realize more robust and precise localization. The open-source KITTI dataset and field tests are used to test the proposed method. The method effectiveness shown in the test results achieves 5-10 cm mapping accuracy and 20-30 cm localization accuracy, and it realizes online autonomous driving in complex scenarios.
Collapse
|
43
|
A Review of Common Techniques for Visual Simultaneous Localization and Mapping. JOURNAL OF ROBOTICS 2023. [DOI: 10.1155/2023/8872822] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/19/2023]
Abstract
Mobile robots are widely used in medicine, agriculture, home furnishing, and industry. Simultaneous localization and mapping (SLAM) is the working basis of mobile robots, so it is extremely necessary and meaningful for making researches on SLAM technology. SLAM technology involves robot mechanism kinematics, logic, mathematics, perceptual detection, and other fields. However, it faces the problem of classifying the technical content, which leads to diverse technical frameworks of SLAM. Among all sorts of SLAM, visual SLAM (V-SLAM) has become the key academic research due to its advantages of low price, easy installation, and simple algorithm model. Firstly, we illustrate the superiority of V-SLAM by comparing it with other localization techniques. Secondly, we sort out some open-source V-SLAM algorithms and compare their real-time performance, robustness, and innovation. Then, we analyze the frameworks, mathematical models, and related basic theoretical knowledge of V-SLAM. Meanwhile, we review the related works from four aspects: visual odometry, back-end optimization, loop closure detection, and mapping. Finally, we prospect the future development trend and make a foundation for researchers to expand works in the future. All in all, this paper classifies each module of V-SLAM in detail and provides better readability to readers. This is undoubtedly the most comprehensive review of V-SLAM recently.
Collapse
|
44
|
Liu Z, Shi D, Li R, Yang S. ESVIO: Event-Based Stereo Visual-Inertial Odometry. SENSORS (BASEL, SWITZERLAND) 2023; 23:1998. [PMID: 36850602 PMCID: PMC9961954 DOI: 10.3390/s23041998] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 01/30/2023] [Accepted: 02/08/2023] [Indexed: 06/18/2023]
Abstract
The emerging event cameras are bio-inspired sensors that can output pixel-level brightness changes at extremely high rates, and event-based visual-inertial odometry (VIO) is widely studied and used in autonomous robots. In this paper, we propose an event-based stereo VIO system, namely ESVIO. Firstly, we present a novel direct event-based VIO method, which fuses events' depth, Time-Surface images, and pre-integrated inertial measurement to estimate the camera motion and inertial measurement unit (IMU) biases in a sliding window non-linear optimization framework, effectively improving the state estimation accuracy and robustness. Secondly, we design an event-inertia semi-joint initialization method, through two steps of event-only initialization and event-inertia initial optimization, to rapidly and accurately solve the initialization parameters of the VIO system, thereby further improving the state estimation accuracy. Based on these two methods, we implement the ESVIO system and evaluate the effectiveness and robustness of ESVIO on various public datasets. The experimental results show that ESVIO achieves good performance in both accuracy and robustness when compared with other state-of-the-art event-based VIO and stereo visual odometry (VO) systems, and, at the same time, with no compromise to real-time performance.
Collapse
Affiliation(s)
- Zhe Liu
- College of Computer, National University of Defense Technology, Changsha 410005, China
| | - Dianxi Shi
- Artificial Intelligence Research Center (AIRC), Defense Innovation Institute, Beijing 100166, China
| | - Ruihao Li
- Artificial Intelligence Research Center (AIRC), Defense Innovation Institute, Beijing 100166, China
| | - Shaowu Yang
- College of Computer, National University of Defense Technology, Changsha 410005, China
| |
Collapse
|
45
|
The Cloud-to-Edge-to-IoT Continuum as an Enabler for Search and Rescue Operations. FUTURE INTERNET 2023. [DOI: 10.3390/fi15020055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/31/2023] Open
Abstract
When a natural or human disaster occurs, time is critical and often of vital importance. Data from the incident area containing the information to guide search and rescue (SAR) operations and improve intervention effectiveness should be collected as quickly as possible and with the highest accuracy possible. Nowadays, rescuers are assisted by different robots able to fly, climb or crawl, and with different sensors and wireless communication means. However, the heterogeneity of devices and data together with the strong low-delay requirements cause these technologies not yet to be used at their highest potential. Cloud and Edge technologies have shown the capability to offer support to the Internet of Things (IoT), complementing it with additional resources and functionalities. Nonetheless, building a continuum from the IoT to the edge and to the cloud is still an open challenge. SAR operations would benefit strongly from such a continuum. Distributed applications and advanced resource orchestration solutions over the continuum in combination with proper software stacks reaching out to the edge of the network may enhance the response time and effective intervention for SAR operation. The challenges for SAR operations, the technologies, and solutions for the cloud-to-edge-to-IoT continuum will be discussed in this paper.
Collapse
|
46
|
Levine S, Shah D. Learning robotic navigation from experience: principles, methods and recent results. Philos Trans R Soc Lond B Biol Sci 2023; 378:20210447. [PMID: 36511408 PMCID: PMC9745865 DOI: 10.1098/rstb.2021.0447] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Accepted: 06/13/2022] [Indexed: 12/15/2022] Open
Abstract
Navigation is one of the most heavily studied problems in robotics and is conventionally approached as a geometric mapping and planning problem. However, real-world navigation presents a complex set of physical challenges that defies simple geometric abstractions. Machine learning offers a promising way to go beyond geometry and conventional planning, allowing for navigational systems that make decisions based on actual prior experience. Such systems can reason about traversability in ways that go beyond geometry, accounting for the physical outcomes of their actions and exploiting patterns in real-world environments. They can also improve as more data is collected, potentially providing a powerful network effect. In this article, we present a general toolkit for experiential learning of robotic navigation skills that unifies several recent approaches, describe the underlying design principles, summarize experimental results from several of our recent papers, and discuss open problems and directions for future work. This article is part of the theme issue 'New approaches to 3D vision'.
Collapse
Affiliation(s)
- Sergey Levine
- Berkeley AI Research (BAIR), UC Berkeley, 2121 Berkeley Way, Berkeley, CA 94704, USA
| | - Dhruv Shah
- Berkeley AI Research (BAIR), UC Berkeley, 2121 Berkeley Way, Berkeley, CA 94704, USA
| |
Collapse
|
47
|
Xiang G, Dian S, Zhao N, Wang G. Semantic-Structure-Aware Multi-Level Information Fusion for Robust Global Orientation Optimization of Autonomous Mobile Robots. SENSORS (BASEL, SWITZERLAND) 2023; 23:1125. [PMID: 36772164 PMCID: PMC9920800 DOI: 10.3390/s23031125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/26/2022] [Revised: 01/12/2023] [Accepted: 01/13/2023] [Indexed: 06/18/2023]
Abstract
Multi-camera-based simultaneous localization and mapping (SLAM) has been widely applied in various mobile robots under uncertain or unknown environments to accomplish tasks autonomously. However, the conventional purely data-driven feature extraction methods cannot utilize the rich semantic information in the environment, which leads to the performance of the SLAM system being susceptible to various interferences. In this work, we present a semantic-aware multi-level information fusion scheme for robust global orientation estimation. Specifically, a visual semantic perception system based on the synthesized surround view image is proposed for the multi-eye surround vision system widely used in mobile robots, which is used to obtain the visual semantic information required for SLAM tasks. The original multi-eye image was first transformed to the synthesized surround view image, and the passable space was extracted with the help of the semantic segmentation network model as a mask for feature extraction; moreover, the hybrid edge information was extracted to effectively eliminate the distorted edges by further using the distortion characteristics of the reverse perspective projection process. Then, the hybrid semantic information was used for robust global orientation estimation; thus, better localization performance was obtained. The experiments on an intelligent vehicle, which was used for automated valet parking both in indoor and outdoor scenes, showed that the proposed hybrid multi-level information fusion method achieved at least a 10-percent improvement in comparison with other edge segmentation methods, the average orientation estimation error being between 1 and 2 degrees, much smaller than other methods, and the trajectory drift value of the proposed method was much smaller than that of other methods.
Collapse
Affiliation(s)
- Guofei Xiang
- College of Electrical Engineering, Sichuan University, Chengdu 610065, China
- National Key Laboratory of Special Vehicle Design and Manufacturing Integration Technology, Baotou 014031, China
| | - Songyi Dian
- College of Electrical Engineering, Sichuan University, Chengdu 610065, China
| | - Ning Zhao
- National Key Laboratory of Special Vehicle Design and Manufacturing Integration Technology, Baotou 014031, China
| | - Guodong Wang
- National Key Laboratory of Special Vehicle Design and Manufacturing Integration Technology, Baotou 014031, China
| |
Collapse
|
48
|
Fan G, Huang J, Yang D, Rao L. Sampling visual SLAM with a wide‐angle camera for legged mobile robots. IET CYBER-SYSTEMS AND ROBOTICS 2023. [DOI: 10.1049/csy2.12074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/06/2023] Open
Affiliation(s)
- Guangyu Fan
- School of Electronic Information Shanghai DianJi University Shanghai China
| | - Jiaxin Huang
- School of Electronic Information Shanghai DianJi University Shanghai China
| | | | - Lei Rao
- School of Electronic Information Shanghai DianJi University Shanghai China
| |
Collapse
|
49
|
Legittimo M, Felicioni S, Bagni F, Tagliavini A, Dionigi A, Gatti F, Verucchi M, Costante G, Bertogna M. A benchmark analysis of data‐driven and geometric approaches for robot ego‐motion estimation. J FIELD ROBOT 2023. [DOI: 10.1002/rob.22151] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
Affiliation(s)
- Marco Legittimo
- Department of Engineering University of Perugia Perugia Italy
| | | | - Fabio Bagni
- Department of Physics, Informatics and Mathematics University of Modena and Reggio Emilia Modena Italy
- Hipert S.r.l. Modena Italy
| | | | - Alberto Dionigi
- Department of Engineering University of Perugia Perugia Italy
| | | | - Micaela Verucchi
- Department of Physics, Informatics and Mathematics University of Modena and Reggio Emilia Modena Italy
| | | | - Marko Bertogna
- Department of Physics, Informatics and Mathematics University of Modena and Reggio Emilia Modena Italy
- Hipert S.r.l. Modena Italy
| |
Collapse
|
50
|
Droukas L, Doulgeri Z, Tsakiridis NL, Triantafyllou D, Kleitsiotis I, Mariolis I, Giakoumis D, Tzovaras D, Kateris D, Bochtis D. A Survey of Robotic Harvesting Systems and Enabling Technologies. J INTELL ROBOT SYST 2023; 107:21. [PMID: 36721646 PMCID: PMC9881528 DOI: 10.1007/s10846-022-01793-z] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 11/25/2022] [Indexed: 01/28/2023]
Abstract
This paper presents a comprehensive review of ground agricultural robotic systems and applications with special focus on harvesting that span research and commercial products and results, as well as their enabling technologies. The majority of literature concerns the development of crop detection, field navigation via vision and their related challenges. Health monitoring, yield estimation, water status inspection, seed planting and weed removal are frequently encountered tasks. Regarding robotic harvesting, apples, strawberries, tomatoes and sweet peppers are mainly the crops considered in publications, research projects and commercial products. The reported harvesting agricultural robotic solutions, typically consist of a mobile platform, a single robotic arm/manipulator and various navigation/vision systems. This paper reviews reported development of specific functionalities and hardware, typically required by an operating agricultural robot harvester; they include (a) vision systems, (b) motion planning/navigation methodologies (for the robotic platform and/or arm), (c) Human-Robot-Interaction (HRI) strategies with 3D visualization, (d) system operation planning & grasping strategies and (e) robotic end-effector/gripper design. Clearly, automated agriculture and specifically autonomous harvesting via robotic systems is a research area that remains wide open, offering several challenges where new contributions can be made.
Collapse
Affiliation(s)
- Leonidas Droukas
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki (AUTH), Thessaloniki, 54124 Greece
| | - Zoe Doulgeri
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki (AUTH), Thessaloniki, 54124 Greece
| | - Nikolaos L. Tsakiridis
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki (AUTH), Thessaloniki, 54124 Greece
| | - Dimitra Triantafyllou
- Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Thessaloniki, 57001 Greece
| | - Ioannis Kleitsiotis
- Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Thessaloniki, 57001 Greece
| | - Ioannis Mariolis
- Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Thessaloniki, 57001 Greece
| | - Dimitrios Giakoumis
- Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Thessaloniki, 57001 Greece
| | - Dimitrios Tzovaras
- Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Thessaloniki, 57001 Greece
| | - Dimitrios Kateris
- Institute for Bio-Economy and Agri-Technology (iBO), Centre for Research and Technology Hellas (CERTH), Volos, 38333 Greece
| | - Dionysis Bochtis
- Institute for Bio-Economy and Agri-Technology (iBO), Centre for Research and Technology Hellas (CERTH), Volos, 38333 Greece
| |
Collapse
|