Introducing Symmetries to Black Box Meta Reinforcement Learning

Authors

  • Louis Kirsch The Swiss AI Lab IDSIA
  • Sebastian Flennerhag DeepMind
  • Hado van Hasselt DeepMind
  • Abram Friesen DeepMind
  • Junhyuk Oh DeepMind
  • Yutian Chen DeepMind

DOI:

https://doi.org/10.1609/aaai.v36i7.20681

Keywords:

Machine Learning (ML)

Abstract

Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction. In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network. These methods are very flexible, but they tend to underperform compared to human-engineered RL algorithms in terms of generalisation to new, unseen environments. In this paper, we explore the role of symmetries in meta-generalisation. We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (specifically the reuse of the learning rule, and invariance to input and output permutations) that are not present in typical black-box meta RL systems. We hypothesise that these symmetries can play an important role in meta-generalisation. Building off recent work in black-box supervised meta learning, we develop a black-box meta RL system that exhibits these same symmetries. We show through careful experimentation that incorporating these symmetries can lead to algorithms with a greater ability to generalise to unseen action & observation spaces, tasks, and environments.

Downloads

Published

2022-06-28

How to Cite

Kirsch, L., Flennerhag, S., Hasselt, H. . . van, Friesen, A., Oh, J., & Chen, Y. (2022). Introducing Symmetries to Black Box Meta Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7202-7210. https://doi.org/10.1609/aaai.v36i7.20681

Issue

Section

AAAI Technical Track on Machine Learning II