Deep Deterministic Policy Gradient-Based Autonomous Driving for Mobile Robots in Sparse Reward Environments

Inspection Robot Navigation Based on Improved TD3 Algorithm

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long-short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics.

Adaptive Control for Virtual Synchronous Generator Parameters Based on Soft Actor Critic

This paper introduces a model-free optimization method based on reinforcement learning (RL) aimed at resolving the issues of active power and frequency oscillations present in a traditional virtual synchronous generator (VSG). The RL agent utilizes the active power and frequency response of the VSG as state information inputs and generates actions to adjust the virtual inertia and damping coefficients for an optimal response. Distinctively, this study incorporates a setting-time term into the reward function design, alongside power and frequency deviations, to avoid prolonged system transients due to over-optimization. The soft actor critic (SAC) algorithm is utilized to determine the optimal strategy. SAC, being model-free with fast convergence, avoids policy overestimation bias, thus achieving superior convergence results. Finally, the proposed method is validated through MATLAB/Simulink simulation. Compared to other approaches, this method more effectively suppresses oscillations in active power and frequency and significantly reduces the setting time.

Enhancing photovoltaic parameter estimation: integration of non-linear hunting and reinforcement learning strategies with golden jackal optimizer

The advancement of Photovoltaic (PV) systems hinges on the precise optimization of their parameters. Among the numerous optimization techniques, the effectiveness of each often rests on their inherent parameters. This research introduces a new methodology, the Reinforcement Learning-based Golden Jackal Optimizer (RL-GJO). This approach uniquely combines reinforcement learning with the Golden Jackal Optimizer to enhance its efficiency and adaptability in handling various optimization problems. Furthermore, the research incorporates an advanced non-linear hunting strategy to optimize the algorithm's performance. The proposed algorithm is first validated using 29 CEC2017 benchmark test functions and five engineering-constrained design problems. Secondly, rigorous testing on PV parameter estimation benchmark datasets, including the single-diode model, double-diode model, three-diode model, and a representative PV module, was carried out to highlight the superiority of RL-GJO. The results were compelling: the root mean square error values achieved by RL-GJO were markedly lower than those of the original algorithm and other prevalent optimization methods. The synergy between reinforcement learning and GJO in this approach facilitates faster convergence and improved solution quality. This integration not only improves the performance metrics but also ensures a more efficient optimization process, especially in complex PV scenarios. With an average Freidman's rank test values of 1.564 for numerical and engineering design problems and 1.742 for parameter estimation problems, the proposed RL-GJO is performing better than the original GJO and other peers. The proposed RL-GJO stands out as a reliable tool for PV parameter estimation. By seamlessly combining reinforcement learning with the golden jackal optimizer, it sets a new benchmark in PV optimization, indicating a promising avenue for future research and applications.

Autonomous Driving of Mobile Robots in Dynamic Environments Based on Deep Deterministic Policy Gradient: Reward Shaping and Hindsight Experience Replay

In this paper, we propose a reinforcement learning-based end-to-end learning method for the autonomous driving of a mobile robot in a dynamic environment with obstacles. Applying two additional techniques for reinforcement learning simultaneously helps the mobile robot in finding an optimal policy to reach the destination without collisions. First, the multifunctional reward-shaping technique guides the agent toward the goal by utilizing information about the destination and obstacles. Next, employing the hindsight experience replay technique to address the experience imbalance caused by the sparse reward problem assists the agent in finding the optimal policy. We validated the proposed technique in both simulation and real-world environments. To assess the effectiveness of the proposed method, we compared experiments for five different cases.

End-to-End Autonomous Navigation Based on Deep Reinforcement Learning with a Survival Penalty Function

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor-critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a nonholonomic wheeled mobile robot (WMR) to perform navigation in dynamic environments containing obstacles and for which no maps are available. A comprehensive reward based on the survival penalty function is introduced; this approach effectively solves the sparse reward problem and enables the WMR to move toward its target. Consecutive episodes are connected to increase the cumulative penalty for scenarios involving obstacles; this method prevents training failure and enables the WMR to plan a collision-free path. Simulations are conducted for four scenarios-movement in an obstacle-free space, in a parking lot, at an intersection without and with a central obstacle, and in a multiple obstacle space-to demonstrate the efficiency and operational safety of our method. For the same navigation environment, compared with the DDPG algorithm, the TD3 algorithm exhibits faster numerical convergence and higher stability in the training phase, as well as a higher task execution success rate in the evaluation phase.