Adversarial Self-Attention for Language Understanding
Keywords:SNLP: Adversarial Attacks & Robustness, SNLP: Language Models
AbstractDeep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose Adversarial Self-Attention mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gain compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.
How to Cite
Wu, H., Ding, R., Zhao, H., Xie, P., Huang, F., & Zhang, M. (2023). Adversarial Self-Attention for Language Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13727-13735. https://doi.org/10.1609/aaai.v37i11.26608
AAAI Technical Track on Speech & Natural Language Processing