ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

Authors

  • Huiqun Li Shandong University
  • Hanhan Zhou The George Washington University
  • Yifei Zou Shandong University
  • Dongxiao Yu Shandong University
  • Tian Lan The George Washington University

DOI:

https://doi.org/10.1609/aaai.v38i16.29695

Keywords:

MAS: Multiagent Learning, MAS: Agent/AI Theories and Architectures

Abstract

Value function factorization has achieved great success in multi-agent reinforcement learning by optimizing joint action-value functions through the maximization of factorized per-agent utilities. To ensure Individual-Global-Maximum property, existing works often focus on value factorization using monotonic functions, which are known to result in restricted representation expressiveness. In this paper, we analyze the limitations of monotonic factorization and present ConcaveQ, a novel non-monotonic value function factorization approach that goes beyond monotonic mixing functions and employs neural network representations of concave mixing functions. Leveraging the concave property in factorization, an iterative action selection scheme is developed to obtain optimal joint actions during training. It is used to update agents’ local policy networks, enabling fully decentralized execution. The effectiveness of the proposed ConcaveQ is validated across scenarios involving multi-agent predator-prey environment and StarCraft II micromanagement tasks. Empirical results exhibit significant improvement of ConcaveQ over state-of-the-art multi-agent reinforcement learning approaches.

Published

2024-03-24

How to Cite

Li, H., Zhou, H., Zou, Y., Yu, D., & Lan, T. (2024). ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 17461-17468. https://doi.org/10.1609/aaai.v38i16.29695

Issue

Section

AAAI Technical Track on Multiagent Systems