GOHSP: A Unified Framework of Graph and Optimization-Based Heterogeneous Structured Pruning for Vision Transformer - Yin, Miao; Uzkent, Burak; Shen, Yilin; Jin, Hongxia; Yuan, Bo (Proceedings of the AAAI Conference on Artificial Intelligence, 2023)
The recently proposed Vision transformers (ViTs) have shownvery impressive empirical performance in various computer vision tasks,and they are viewed as an important type of foundation model. However, ViTs are typically constructed with large-scale sizes, which thenseverely hinder their potential deployment in many practical resources constrained applications. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enablepractical efficiency. However, unlike its current popularity for CNNs andRNNs, structured pruning for ViT models is little explored.In this paper, we propose GOHSP, a unified framework of Graph andOptimization-based Structured Pruning for ViT models. We first developa graph-based ranking for measuring the importance of attention heads,and the extracted importance information is further integrated to anoptimization-based procedure to impose the heterogeneous structuredsparsity patterns on the ViT models. Experimental results show thatour proposed GOHSP demonstrates excellent compression performance.On CIFAR-10 dataset, our approach can bring 40% parameters reductionwith no accuracy loss for ViT-Small model. On ImageNet dataset, with30% and 35% sparsity ratio for DeiT-Tiny and DeiT-Small models, ourapproach achieves 1.65% and 0.76% accuracy increase over the existingstructured pruning methods, respectively.