Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes


  • Tomáš Brázdil Masaryk University
  • Krishnendu Chatterjee Institute of Science and Technology Austria
  • Petr Novotný Masaryk University
  • Jiří Vahala Masaryk University



Markov decision processes (MDPs) are the defacto framework for sequential decision making in the presence of stochastic uncertainty. A classical optimization criterion for MDPs is to maximize the expected discounted-sum payoff, which ignores low probability catastrophic events with highly negative impact on the system. On the other hand, risk-averse policies require the probability of undesirable events to be below a given threshold, but they do not account for optimization of the expected payoff. We consider MDPs with discounted-sum payoff with failure states which represent catastrophic outcomes. The objective of risk-constrained planning is to maximize the expected discounted-sum payoff among risk-averse policies that ensure the probability to encounter a failure state is below a desired threshold. Our main contribution is an efficient risk-constrained planning algorithm that combines UCT-like search with a predictor learned through interaction with the MDP (in the style of AlphaZero) and with a risk-constrained action selection via linear programming. We demonstrate the effectiveness of our approach with experiments on classical MDPs from the literature, including benchmarks with an order of 106 states.




How to Cite

Brázdil, T., Chatterjee, K., Novotný, P., & Vahala, J. (2020). Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 34(06), 9794-9801.



AAAI Technical Track: Planning, Routing, and Scheduling