Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT


  • Sheng Shen Univerisity of California, Berkeley
  • Zhen Dong Univerisity of California, Berkeley
  • Jiayu Ye Univerisity of California, Berkeley
  • Linjian Ma Univerisity of California, Berkeley
  • Zhewei Yao Univerisity of California, Berkeley
  • Amir Gholami Univerisity of California, Berkeley
  • Michael W. Mahoney Univerisity of California, Berkeley
  • Kurt Keutzer Univerisity of California, Berkeley




Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use Hessian-based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.




How to Cite

Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020). Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8815-8821. https://doi.org/10.1609/aaai.v34i05.6409



AAAI Technical Track: Natural Language Processing