TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL

Authors

  • Yanchao Sun University of Maryland, College Park
  • Xiangyu Yin Beijing University of Posts and Telecommunications
  • Furong Huang University of Maryland, College Park

Keywords:

Reinforcement Learning, Transfer/Adaptation/Multi-task/Meta/Automated Learning

Abstract

Transferring knowledge among various environments is important for efficiently learning multiple tasks online. Most existing methods directly use the previously learned models or previously learned optimal policies to learn new tasks. However, these methods may be inefficient when the underlying models or optimal policies are substantially different across tasks. In this paper, we propose Template Learning (TempLe), a PAC-MDP method for multi-task reinforcement learning that could be applied to tasks with varying state/action space without prior knowledge of inter-task mappings. TempLe gains sample efficiency by extracting similarities of the transition dynamics across tasks even when their underlying models or optimal policies have limited commonalities. We present two algorithms for an ``online'' and a ``finite-model'' setting respectively. We prove that our proposed TempLe algorithms achieve much lower sample complexity than single-task learners or state-of-the-art multi-task methods. We show via systematically designed experiments that our TempLe method universally outperforms the state-of-the-art multi-task methods (PAC-MDP or not) in various settings and regimes.

Downloads

Published

2021-05-18

How to Cite

Sun, Y., Yin, X., & Huang, F. (2021). TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11), 9765-9773. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17174

Issue

Section

AAAI Technical Track on Machine Learning IV