TY - JOUR
AU - Brázdil, Tomáš
AU - Chatterjee, Krishnendu
AU - Novotný, Petr
AU - Vahala, Jiří
PY - 2020/04/03
Y2 - 2021/05/17
TI - Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes
JF - Proceedings of the AAAI Conference on Artificial Intelligence
JA - AAAI
VL - 34
IS - 06
SE - AAAI Technical Track: Planning, Routing, and Scheduling
DO - 10.1609/aaai.v34i06.6531
UR - https://ojs.aaai.org/index.php/AAAI/article/view/6531
SP - 9794-9801
AB - <p>Markov decision processes (MDPs) are the defacto framework for sequential decision making in the presence of stochastic uncertainty. A classical optimization criterion for MDPs is to maximize the expected discounted-sum payoff, which ignores low probability catastrophic events with highly negative impact on the system. On the other hand, risk-averse policies require the probability of undesirable events to be below a given threshold, but they do not account for optimization of the expected payoff. We consider MDPs with discounted-sum payoff with failure states which represent catastrophic outcomes. The objective of <em>risk-constrained</em> planning is to maximize the expected discounted-sum payoff among risk-averse policies that ensure the probability to encounter a failure state is below a desired threshold. Our main contribution is an efficient risk-constrained planning algorithm that combines UCT-like search with a predictor learned through interaction with the MDP (in the style of AlphaZero) and with a risk-constrained action selection via linear programming. We demonstrate the effectiveness of our approach with experiments on classical MDPs from the literature, including benchmarks with an order of 10<sup>6</sup> states.</p>
ER -