ELMA: Energy-Based Learning for Multi-Agent Activity Forecasting


  • Yuke Li UC Berkeley
  • Pin Wang University of California, Berkeley
  • Lixiong Chen University of Oxford
  • Zheng Wang Wuhan University
  • Ching-Yao Chan UC Berkeley




Computer Vision (CV)


This paper describes an energy-based learning method that predicts the activities of multiple agents simultaneously. It aims to forecast both upcoming actions and paths of all agents in a scene based on their past activities, which can be jointly formulated by a probabilistic model over time. Learning this model is challenging because: 1) it has a large number of time-dependent variables that must scale with the forecast horizon and the number of agents; 2) distribution functions have to contain multiple modes in order to capture the spatio-temporal complexities of each agent's activities. To address these challenges, we put forth a novel Energy-based Learning approach for Multi-Agent activity forecasting (ELMA) to estimate this complex model via maximum log-likelihood estimation. Specifically, by sampling from a sequence of factorized marginalized multi-model distributions, ELMA generates most possible future actions efficiently. Moreover, by graph-based representations, ELMA also explicitly resolves the spatio-temporal dependencies of all agents' activities in a single pass. Our experiments on two large-scale datasets prove that ELMA outperforms recent leading studies by an obvious margin.




How to Cite

Li, Y., Wang, P., Chen, L., Wang, Z., & Chan, C.-Y. (2022). ELMA: Energy-Based Learning for Multi-Agent Activity Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), 1482-1490. https://doi.org/10.1609/aaai.v36i2.20038



AAAI Technical Track on Computer Vision II