Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning

Authors

  • Jacob Adamczyk University of Massachusetts Boston
  • Argenis Arriojas University of Massachusetts Boston
  • Stas Tiomkin San Jose State University
  • Rahul V. Kulkarni University of Massachusetts Boston

DOI:

https://doi.org/10.1609/aaai.v37i6.25817

Keywords:

ML: Reinforcement Learning Theory, ML: Reinforcement Learning Algorithms, ML: Lifelong and Continual Learning

Abstract

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

Downloads

Published

2023-06-26

How to Cite

Adamczyk, J., Arriojas, A., Tiomkin, S., & Kulkarni, R. V. (2023). Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 6658-6665. https://doi.org/10.1609/aaai.v37i6.25817

Issue

Section

AAAI Technical Track on Machine Learning I