Toward a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Control

Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm. The energy function is designed to increase rapidly for potentially dangerous actions, locating the safe set on the action space. Therefore, we can identify the dangerous actions prior to taking them and achieve zero-constraint violation. Our major contributions are twofold. First, we use the data-driven methods to learn the energy function, which releases the requirement of known dynamics. Second, we formulate a constrained RL problem to solve the zero-violation policies. We prove that our Lagrangian-based constrained RL solutions converge to the constrained optimal zero-violation policies theoretically. The proposed algorithm is evaluated on the complex simulation environments and a hardware-in-loop (HIL) experiment with a real autonomous vehicle controller. Experimental results suggest that the converged policies in all environments achieve zero-constraint violation and comparable performance with model-based baseline.

Reinforcement learning algorithm for improving speed response of a five-phase permanent magnet synchronous motor based model predictive control

Enhancing the performance of 5ph-IPMSM control plays a crucial role in advancing various innovative applications such as electric vehicles. This paper proposes a new reinforcement learning (RL) control algorithm based twin-delayed deep deterministic policy gradient (TD3) algorithm to tune two cascaded PI controllers in a five-phase interior permanent magnet synchronous motor (5ph-IPMSM) drive system based model predictive control (MPC). The main purpose of the control methodology is to optimize the 5ph-IPMSM speed response either in constant torque region or constant power region. The speed responses obtained using RL control algorithm are compared with those obtained using four of the most recent metaheuristic optimization techniques (MHOT) which are Transit Search (TS), Honey Badger Algorithm (HBA), Dwarf Mongoose (DM), and Dandelion-Optimizer (DO) optimization techniques. The speed response are compared in terms of the settling time, rise time, maximum time and maximum overshoot percentage. It is found that the suggested RL based TD3 give minimum settling time and relatively low values for the rise time, max time and overshoot percentage which makes the RL provide superior speed responses compared with those obtained from the four MHOT. The drive system speed responses are obtained in the constant torque region and constant power region using MATLAB SIMULINK package.

Machine Learning for Design Optimization of Electromagnetic Devices: Recent Developments and Future Directions

This paper reviews the recent developments of design optimization methods for electromagnetic devices, with a focus on machine learning methods. First, the recent advances in multi-objective, multidisciplinary, multilevel, topology, fuzzy, and robust design optimization of electromagnetic devices are overviewed. Second, a review is presented to the performance prediction and design optimization of electromagnetic devices based on the machine learning algorithms, including artificial neural network, support vector machine, extreme learning machine, random forest, and deep learning. Last, to meet modern requirements of high manufacturing/production quality and lifetime reliability, several promising topics, including the application of cloud services and digital twin, are discussed as future directions for design optimization of electromagnetic devices.