Cross-Lingual Natural Language Generation via Pre-Training


  • Zewen Chi Beijing Institute of Technology
  • Li Dong Microsoft Research Asia
  • Furu Wei Microsoft Research Asia
  • Wenhui Wang Microsoft Research Asia
  • Xian-Ling Mao Beijing Institute of Technology
  • Heyan Huang Beijing Institute of technology



In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at




How to Cite

Chi, Z., Dong, L., Wei, F., Wang, W., Mao, X.-L., & Huang, H. (2020). Cross-Lingual Natural Language Generation via Pre-Training. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7570-7577.



AAAI Technical Track: Natural Language Processing