Near Lossless Transfer Learning for Spiking Neural Networks

Authors

  • Zhanglu Yan National University of Singapore
  • Jun Zhou National University of Singapore
  • Weng-Fai Wong National University of Singapore

Keywords:

(Deep) Neural Network Algorithms

Abstract

Spiking neural networks (SNNs) significantly reduce energy consumption by replacing weight multiplications with additions. This makes SNNs suitable for energy-constrained platforms. However, due to its discrete activation, training of SNNs remains a challenge. A popular approach is to first train an equivalent CNN using traditional backpropagation, and then transfer the weights to the intended SNN. Unfortunately, this often results in significant accuracy loss, especially in deeper networks. In this paper, we propose CQ training (Clamped and Quantized training), an SNN-compatible CNN training algorithm with clamp and quantization that achieves near-zero conversion accuracy loss. Essentially, CNN training in CQ training accounts for certain SNN characteristics. Using a 7 layer VGG-* and a 21 layer VGG-19, running on the CIFAR-10 dataset, we achieved 94.16% and 93.44% accuracy in the respective equivalent SNNs. It outperforms other existing comparable works that we know of. We also demonstrate the low-precision weight compatibility for the VGG-19 structure. Without retraining, an accuracy of 93.43% and 92.82% using quantized 9-bit and 8-bit weights, respectively, was achieved. The framework was developed in PyTorch and is publicly available.

Downloads

Published

2021-05-18

How to Cite

Yan, Z., Zhou, J., & Wong, W.-F. (2021). Near Lossless Transfer Learning for Spiking Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12), 10577-10584. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17265

Issue

Section

AAAI Technical Track on Machine Learning V