Improved Text Classification via Contrastive Adversarial Training

Authors

  • Lin Pan IBM
  • Chung-Wei Hang IBM
  • Avirup Sil IBM Research AI
  • Saloni Potdar IBM

DOI:

https://doi.org/10.1609/aaai.v36i10.21362

Keywords:

Speech & Natural Language Processing (SNLP)

Abstract

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during fine-tuning we generate adversarial examples by perturbing the word embedding matrix of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned Bert_Large model outperforms Bert_Large baseline by 1.7% on average, and our fine-tuned Roberta_Large improves over Roberta_Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned Roberta_Large outperforms Roberta_Large baseline by 1-2% on average. For the challenging low-resource scenario, we train our system using half of the training data (per intent) in each of the three intent classification datasets, and achieve similar performance compared to the baseline trained with full training data.

Downloads

Published

2022-06-28

How to Cite

Pan, L., Hang, C.-W., Sil, A., & Potdar, S. (2022). Improved Text Classification via Contrastive Adversarial Training. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11130-11138. https://doi.org/10.1609/aaai.v36i10.21362

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing