Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy

Authors

  • Ramtin Keramati Stanford University
  • Christoph Dann Carnegie Mellon University
  • Alex Tamkin Stanford University
  • Emma Brunskill Standford University

DOI:

https://doi.org/10.1609/aaai.v34i04.5870

Abstract

While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, relatively little is known about how to explore to quickly learn policies with good CVaR. In this paper, we present the first algorithm for sample-efficient learning of CVaR-optimal policies in Markov decision processes based on the optimism in the face of uncertainty principle. This method relies on a novel optimistic version of the distributional Bellman operator that moves probability mass from the lower to the upper tail of the return distribution. We prove asymptotic convergence and optimism of this operator for the tabular policy evaluation case. We further demonstrate that our algorithm finds CVaR-optimal policies substantially faster than existing baselines in several simulated environments with discrete and continuous state spaces.

Downloads

Published

2020-04-03

How to Cite

Keramati, R., Dann, C., Tamkin, A., & Brunskill, E. (2020). Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 4436-4443. https://doi.org/10.1609/aaai.v34i04.5870

Issue

Section

AAAI Technical Track: Machine Learning