Learning and Exploiting Shaped Reward Models for Large Scale Multiagent RL

Authors

  • Arambam James Singh Singapore Management University
  • Akshat Kumar Singapore Management University
  • Hoong Chuin Lau Singapore Management University

Keywords:

Model Representation And Learning Domain Models For Planning, Multi-agent Planning And Learning

Abstract

Many real world systems involve interaction among large number of agents to achieve a common goal, for example, air traffic control. Several model-free RL algorithms have been proposed for such settings. A key limitation is that the empirical reward signal in model-free case is not very effective in addressing the multiagent credit assignment problem, which determines an agent's contribution to the team's success. This results in lower solution quality and high sample complexity. To address this, we contribute (a) an approach to learn a differentiable reward model for both continuous and discrete action setting by exploiting the collective nature of interactions among agents, a feature commonly present in large scale multiagent applications; (b) a shaped reward model analytically derived from the learned reward model to address the key challenge of credit assignment; (c) a model-based multiagent RL approach that integrates shaped rewards into well known RL algorithms such as policy gradient, soft-actor critic. Compared to previous methods, our learned reward models are more accurate, and our approaches achieve better solution quality on synthetic and real world instances of air traffic control, and cooperative navigation with large agent population.

Downloads

Published

2021-05-17

How to Cite

Singh, A. J., Kumar, A., & Lau, H. C. (2021). Learning and Exploiting Shaped Reward Models for Large Scale Multiagent RL. Proceedings of the International Conference on Automated Planning and Scheduling, 31(1), 588-596. Retrieved from https://ojs.aaai.org/index.php/ICAPS/article/view/16007