DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from Scratch

Authors

  • Xiaofeng Ruan National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences
  • Yufan Liu National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences
  • Bing Li National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences PeopleAI Inc.
  • Chunfeng Yuan National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences
  • Weiming Hu National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences CAS Center for Excellence in Brain Science and Intelligence Technology

Keywords:

Object Detection & Categorization

Abstract

Filter pruning is a commonly used method for compressing Convolutional Neural Networks (ConvNets), due to its friendly hardware supporting and flexibility. However, existing methods mostly need a cumbersome procedure, which brings many extra hyper-parameters and training epochs. This is because only using sparsity and pruning stages cannot obtain a satisfying performance. Besides, many works do not consider the difference of pruning ratio across different layers. To overcome these limitations, we propose a novel dynamic and progressive filter pruning (DPFPS) scheme that directly learns a structured sparsity network from Scratch. In particular, DPFPS imposes a new structured sparsity-inducing regularization specifically upon the expected pruning parameters in a dynamic sparsity manner. The dynamic sparsity scheme determines sparsity allocation ratios of different layers and a Taylor series based channel sensitivity criteria is presented to identify the expected pruning parameters. Moreover, we increase the structured sparsity-inducing penalty in a progressive manner. This helps the model to be sparse gradually instead of forcing the model to be sparse at the beginning. Our method solves the pruning ratio based optimization problem by an iterative soft-thresholding algorithm (ISTA) with dynamic sparsity. At the end of the training, we only need to remove the redundant parameters without other stages, such as fine-tuning. Extensive experimental results show that the proposed method is competitive with 11 state-of-the-art methods on both small-scale and large-scale datasets (i.e., CIFAR and ImageNet). Specifically, on ImageNet, we achieve a 44.97% pruning ratio of FLOPs by compressing ResNet-101, even with an increase of 0.12% Top-5 accuracy. Our pruned models and codes are released at https://github.com/taoxvzi/DPFPS.

Downloads

Published

2021-05-18

How to Cite

Ruan, X., Liu, Y., Li, B., Yuan, C., & Hu, W. (2021). DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from Scratch. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 2495-2503. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16351

Issue

Section

AAAI Technical Track on Computer Vision II