ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Authors

  • Jiangrong Shen The College of Computer Science and Technology, Zhejiang University, China
  • Qi Xu School of Artificial Intelligence, Dalian University of Technology, China
  • Jian K. Liu School of Computing, University of Leeds, UK
  • Yueming Wang The College of Computer Science and Technology, Zhejiang University, China
  • Gang Pan The College of Computer Science and Technology, Zhejiang University, China
  • Huajin Tang The College of Computer Science and Technology, Zhejiang University, China Research Institute of Intelligent Computing, Zhejiang Lab, China

DOI:

https://doi.org/10.1609/aaai.v37i1.25079

Keywords:

CMS: Brain Modeling, CMS: Agent & Cognitive Architectures, CMS: Structural Learning and Knowledge Capture

Abstract

Spiking neural networks (SNNs) have manifested remarkable advantages in power consumption and event-driven property during the inference process. To take full advantage of low power consumption and improve the efficiency of these models further, the pruning methods have been explored to find sparse SNNs without redundancy connections after training. However, parameter redundancy still hinders the efficiency of SNNs during training. In the human brain, the rewiring process of neural networks is highly dynamic, while synaptic connections maintain relatively sparse during brain development. Inspired by this, here we propose an efficient evolutionary structure learning (ESL) framework for SNNs, named ESL-SNNs, to implement the sparse SNN training from scratch. The pruning and regeneration of synaptic connections in SNNs evolve dynamically during learning, yet keep the structural sparsity at a certain level. As a result, the ESL-SNNs can search for optimal sparse connectivity by exploring all possible parameters across time. Our experiments show that the proposed ESL-SNNs framework is able to learn SNNs with sparse structures effectively while reducing the limited accuracy. The ESL-SNNs achieve merely 0.28% accuracy loss with 10% connection density on the DVS-Cifar10 dataset. Our work presents a brand-new approach for sparse training of SNNs from scratch with biologically plausible evolutionary mechanisms, closing the gap in the expressibility between sparse training and dense training. Hence, it has great potential for SNN lightweight training and inference with low power consumption and small memory usage.

Downloads

Published

2023-06-26

How to Cite

Shen, J., Xu, Q., Liu, J. K., Wang, Y., Pan, G., & Tang, H. (2023). ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1), 86-93. https://doi.org/10.1609/aaai.v37i1.25079

Issue

Section

AAAI Technical Track on Cognitive Modeling & Cognitive Systems