Predicting Human Mobility with Reinforcement-Learning-Based Long-Term Periodicity Modeling

Runtime verification in uncertain environment based on probabilistic model learning

Runtime verification (RV) is a lightweight approach to detecting temporal errors of system at runtime. It confines the verification on observed trajectory which avoids state explosion problem. To predict the future violation, some work proposed the predictive RV which uses the information from models or static analysis. But for software whose models and codes cannot be obtained, or systems running under uncertain environment, these predictive methods cannot take effect. Meanwhile, RV in general takes multi-valued logic as the specification languages, for example the true, false and inconclusive in three-valued semantics. They cannot give accurate quantitative description of correctness when inconclusive is encountered. We in this paper present a RV method which learns probabilistic model of system and environment from history traces and then generates probabilistic runtime monitor to quantitatively predict the satisfaction of temporal property at each runtime state. In this approach, Hidden Markov Model (HMM) is firstly learned and then transformed to Discrete Time Markov Chain (DTMC). To construct incremental monitor, the monitored LTL property is translated into Deterministic Rabin Automaton (DRA). The final probabilistic monitor is obtained by generating the product of DTMC and DRA, and computing the probabilities for each state. With such a method, one can give early warning once the probability of correctness is lower than a pre-defined threshold, and have the chance to do adjustment in advance. The method has been implemented and experimented on real UAS (Unmanned Aerial Vehicle) simulation platform.

Hierarchical Multi-agent Model for Reinforced MedicalResource Allocation with Imperfect Information

Facing the outbreak of COVID-19, shortage in medical resources becomes increasingly outstanding. Therefore, efficient strategies for medical resource allocation are urgently called for. However, conventional rule-based methods from public health experts have limited capability in dealing with the complex and dynamic pandemic spreading situation. Besides, model-based optimization methods such as dynamic programming (DP) fails to work since we cannot obtain a precise model in the real-world situation most of the time. On the other hand, model-free reinforcement learning (RL) is powerful for decision making, but three key challenges exist in solving this problem via RL: (1) complex situation and countless choices for decision making in the real world; (2) only imperfect information are available due to the latency of pandemic spreading; (3) limitations on conducting experiments in real-world since we cannot set pandemic outbreaks arbitrarily. In this paper, we propose a hierarchical reinforcement learning framework with several specially designed components. We design a decomposed action space with a corresponding training algorithm to deal with the countless choices and ensure efficient and real time strategies. We design an recurrent neural network based framework to utilize the imperfect information obtained from the environment. We also design a multi-agents voting method, which modifies the decision making process considering the randomness during model training and thus improves the performance. We build a pandemic spreading simulator based on real world data, serving as the experimental platform. We conduct extensive experiments and the results show that our method outperforms all the baselines, which reduces infections and deaths by 14.25% on average without the multi-agents voting method and up to 15.44% with it.