Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning


  • Abdus Salam Azad University of California, Berkeley
  • Edward Kim University of California, Berkeley
  • Qiancheng Wu University of California, Berkeley
  • Kimin Lee University of California, Berkeley
  • Ion Stoica University of California, Berkeley
  • Pieter Abbeel University of California, Berkeley
  • Alberto Sangiovanni-Vincentelli University of California, Berkeley
  • Sanjit A. Seshia University of California, Berkeley



Machine Learning (ML)


The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the existing simulators rely on randomly generating the environments based on predefined settings/layouts and offer limited flexibility and control over the environment dynamics for researchers to generate diverse, realistic scenarios as per their demand. To address this issue, for the first time, we formally introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers to model and generate diverse scenarios in an RTS environment in a flexible, systematic, and programmatic manner. To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football (GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities. We also show how researchers/RL practitioners can incorporate their domain knowledge to expedite the training process by intuitively modeling stochastic programmatic policies with SCENIC.




How to Cite

Azad, A. S., Kim, E., Wu, Q., Lee, K., Stoica, I., Abbeel, P., Sangiovanni-Vincentelli, A., & Seshia, S. A. (2022). Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), 6028-6036.



AAAI Technical Track on Machine Learning I