On the Importance of Word Order Information in Cross-lingual Sequence Labeling

Authors

  • Zihan Liu The Hong Kong University of Science and Technology
  • Genta I Winata Hong Kong University of Science and Technology
  • Samuel Cahyawijaya Hong Kong University of Science and Technology
  • Andrea Madotto Hong Kong University of Science and Technology
  • Zhaojiang Lin Hong Kong University of Science and Technology
  • Pascale Fung Hong Kong University of Science and Technology

Keywords:

Machine Translation & Multilinguality, Syntax -- Tagging, Chunking & Parsing

Abstract

Cross-lingual models trained on source language tasks possess the capability to directly transfer to target languages. However, since word order variances generally exist in different languages, cross-lingual models that overfit into the word order of the source language could have sub-optimal performance in target languages. In this paper, we hypothesize that reducing the word order information fitted into the models can improve the adaptation performance in target languages. To verify this hypothesis, we introduce several methods to make models encode less word order information of the source language and test them based on cross-lingual word embeddings and the pre-trained multilingual model. Experimental results on three sequence labeling tasks (i.e., part-of-speech tagging, named entity recognition and slot filling tasks) show that reducing word order information injected into the model can achieve better zero-shot cross-lingual performance. Further analysis illustrates that fitting excessive or insufficient word order information into the model results in inferior cross-lingual performance. Moreover, our proposed methods can also be applied to strong cross-lingual models and further improve their performance.

Downloads

Published

2021-05-18

How to Cite

Liu, Z., Winata, G. I., Cahyawijaya, S., Madotto, A., Lin, Z., & Fung, P. (2021). On the Importance of Word Order Information in Cross-lingual Sequence Labeling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15), 13461-13469. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17588

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing II