HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface

Authors

  • Sungho Shin Seoul National University
  • Jinhwan Park Seoul National University
  • Yoonho Boo Seoul National University
  • Wonyong Sung Seoul National University

DOI:

https://doi.org/10.1609/aaai.v34i04.6035

Abstract

Quantization of deep neural networks is extremely essential for efficient implementations. Low-precision networks are typically designed to represent original floating-point counterparts with high fidelity, and several elaborate quantization algorithms have been developed. We propose a novel training scheme for quantized neural networks to reach flat minima in the loss surface with the aid of quantization noise. The proposed training scheme employs high-low-high-low precision in an alternating manner for network training. The learning rate is also abruptly changed at each stage for coarse- or fine-tuning. With the proposed training technique, we show quite good performance improvements for convolutional neural networks when compared to the previous fine-tuning based quantization scheme. We achieve the state-of-the-art results for recurrent neural network based language modeling with 2-bit weight and activation.

Downloads

Published

2020-04-03

How to Cite

Shin, S., Park, J., Boo, Y., & Sung, W. (2020). HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 5784-5791. https://doi.org/10.1609/aaai.v34i04.6035

Issue

Section

AAAI Technical Track: Machine Learning