Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment

Authors

  • Di Jin MIT
  • Zhijing Jin University of Hong Kong
  • Joey Tianyi Zhou A*STAR
  • Peter Szolovits MIT

DOI:

https://doi.org/10.1609/aaai.v34i05.6311

Abstract

Machine learning algorithms are often vulnerable to adversarial examples that have imperceptible alterations from the original counterparts but can fool the state-of-the-art models. It is helpful to evaluate or even improve the robustness of these models by exposing the maliciously crafted adversarial examples. In this paper, we present TextFooler, a simple but strong baseline to generate adversarial text. By applying it to two fundamental natural language tasks, text classification and textual entailment, we successfully attacked three target models, including the powerful pre-trained BERT, and the widely used convolutional and recurrent neural networks. We demonstrate three advantages of this framework: (1) effective—it outperforms previous attacks by success rate and perturbation rate, (2) utility-preserving—it preserves semantic content, grammaticality, and correct types classified by humans, and (3) efficient—it generates adversarial text with computational complexity linear to the text length.1

Downloads

Published

2020-04-03

How to Cite

Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020). Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8018-8025. https://doi.org/10.1609/aaai.v34i05.6311

Issue

Section

AAAI Technical Track: Natural Language Processing