Towards Making the Most of BERT in Neural Machine Translation


  • Jiacheng Yang Shanghai Jiao Tong University
  • Mingxuan Wang ByteDance AI Lab
  • Hao Zhou ByteDance AI Lab
  • Chengqi Zhao ByteDance AI Lab
  • Weinan Zhang Shanghai Jiao Tong University
  • Yong Yu Shanghai Jiao Tong University
  • Lei Li ByteDance AI Lab



GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTnmt) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTnmt} consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTnmt gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score.




How to Cite

Yang, J., Wang, M., Zhou, H., Zhao, C., Zhang, W., Yu, Y., & Li, L. (2020). Towards Making the Most of BERT in Neural Machine Translation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 9378-9385.



AAAI Technical Track: Natural Language Processing