Hierarchical Average Reward Policy Gradient Algorithms (Student Abstract)


  • Akshay Dharmavaram Birla Institute of Technology and Science, Pilani
  • Matthew Riemer IBM Research
  • Shalabh Bhatnagar Indian Institute of Science, Bangalore




Option-critic learning is a general-purpose reinforcement learning (RL) framework that aims to address the issue of long term credit assignment by leveraging temporal abstractions. However, when dealing with extended timescales, discounting future rewards can lead to incorrect credit assignments. In this work, we address this issue by extending the hierarchical option-critic policy gradient theorem for the average reward criterion. Our proposed framework aims to maximize the long-term reward obtained in the steady-state of the Markov chain defined by the agent's policy. Furthermore, we use an ordinary differential equation based approach for our convergence analysis and prove that the parameters of the intra-option policies, termination functions, and value functions, converge to their corresponding optimal values, with probability one. Finally, we illustrate the competitive advantage of learning options, in the average reward setting, on a grid-world environment with sparse rewards.




How to Cite

Dharmavaram, A., Riemer, M., & Bhatnagar, S. (2020). Hierarchical Average Reward Policy Gradient Algorithms (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 34(10), 13777-13778. https://doi.org/10.1609/aaai.v34i10.7160



Student Abstract Track