AMOM: Adaptive Masking over Masking for Conditional Masked Language Model


  • Yisheng Xiao Soochow University
  • Ruiyang Xu Soochow University
  • Lijun Wu Microsoft Research Asia
  • Juntao Li Soochow University
  • Tao Qin Microsoft Research Asia
  • Tie-Yan Liu Microsoft Research
  • Min Zhang Soochow University



SNLP: Generation, SNLP: Machine Translation & Multilinguality, SNLP: Summarization


Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the inference stage, many non-autoregressive (NAR) strategies have been proposed in the past few years. Among them, the conditional masked language model (CMLM) is one of the most versatile frameworks, as it can support many different sequence generation scenarios and achieve very competitive performance on these tasks. In this paper, we further introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder and make the encoder optimization easier. Experiments on 3 different tasks (neural machine translation, summarization, and code generation) with 15 datasets in total confirm that our proposed simple method achieves significant performance improvement over the strong CMLM model. Surprisingly, our proposed model yields state-of-the-art performance on neural machine translation (34.62 BLEU on WMT16 EN to RO, 34.82 BLEU on WMT16 RO to EN, and 34.84 BLEU on IWSLT De to En) and even better performance than the AR Transformer on 7 benchmark datasets with at least 2.2x speedup. Our code is available at GitHub.




How to Cite

Xiao, Y., Xu, R., Wu, L., Li, J., Qin, T., Liu, T.-Y., & Zhang, M. (2023). AMOM: Adaptive Masking over Masking for Conditional Masked Language Model. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13789-13797.



AAAI Technical Track on Speech & Natural Language Processing