Predictive Auxiliary Learning for Belief-based Multi-Agent Systems

Abstract

Multi-agent reinforcement learning (MARL) under partial observability requires agents to construct reliable belief representations from limited local observations and partial information exchange. Conventional approaches rely primarily on sparse task rewards to shape these internal representations, which often leads to unstable training dynamics and slow convergence. We propose BElief-based Predictive Auxiliary Learning (BEPAL), a decentralized MARL framework that improves belief representation quality through predictive auxiliary tasks. BEPAL trains agents to maintain a "world model" that accurately estimates the global environment state and predict future dynamics from accumulated local observations and communication, providing another channel of feedback that complements sparse reward in reinforcement learning. By encouraging the hidden state to serve as a compact and informative summary of historical input, BEPAL stabilizes decentralized policy learning and accelerates convergence. The proposed approach is compatible with any decentralized MARL architectures with homogeneous agents and does not increase execution-time complexity. Experiments on Predator–Prey, Traffic Junction, Google Research Football, and RWARE show that BEPAL consistently improves learning stability and task performance compared to strong baselines, highlighting the effectiveness of predictive auxiliary learning for belief formation under partial observability.