Learning Compositional Tasks from Language Instructions


  • Lajanugen Logeswaran LG AI Research
  • Wilka Carvalho University of Michigan
  • Honglak Lee LG AI Research University of Michigan




SNLP: Language Grounding, ML: Representation Learning


The ability to combine learned knowledge and skills to solve novel tasks is a key aspect of generalization in humans that allows us to understand and perform tasks described by novel language utterances. While progress has been made in supervised learning settings, no work has yet studied compositional generalization of a reinforcement learning agent following natural language instructions in an embodied environment. We develop a set of tasks in a photo-realistic simulated kitchen environment that allow us to study the degree to which a behavioral policy captures the systematicity in language by studying its zero-shot generalization performance on held out natural language instructions. We show that our agent which leverages a novel additive action-value decomposition in tandem with attention based subgoal prediction is able to exploit composition in text instructions to generalize to unseen tasks.




How to Cite

Logeswaran, L., Carvalho, W., & Lee, H. (2023). Learning Compositional Tasks from Language Instructions. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13300-13308. https://doi.org/10.1609/aaai.v37i11.26561



AAAI Technical Track on Speech & Natural Language Processing