CF-ViT: A General Coarse-to-Fine Method for Vision Transformer
DOI:
https://doi.org/10.1609/aaai.v37i6.25860Keywords:
ML: Deep Neural Architectures, ML: Deep Neural Network AlgorithmsAbstract
Vision Transformers (ViT) have made many breakthroughs in computer vision tasks. However, considerable redundancy arises in the spatial dimension of an input image, leading to massive computational costs. Therefore, We propose a coarse-to-fine vision transformer (CF-ViT) to relieve computational burden while retaining performance in this paper. Our proposed CF-ViT is motivated by two important observations in modern ViT models: (1) The coarse-grained patch splitting can locate informative regions of an input image. (2) Most images can be well recognized by a ViT model in a small-length token sequence. Therefore, our CF-ViT implements network inference in a two-stage manner. At coarse inference stage, an input image is split into a small-length patch sequence for a computationally economical classification. If not well recognized, the informative patches are identified and further re-split in a fine-grained granularity. Extensive experiments demonstrate the efficacy of our CF-ViT. For example, without any compromise on performance, CF-ViT reduces 53% FLOPs of LV-ViT, and also achieves 2.01x throughput. Code of this project is at https://github.com/ChenMnZ/CF-VDownloads
Published
2023-06-26
How to Cite
Chen, M., Lin, M., Li, K., Shen, Y., Wu, Y., Chao, F., & Ji, R. (2023). CF-ViT: A General Coarse-to-Fine Method for Vision Transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 7042-7052. https://doi.org/10.1609/aaai.v37i6.25860
Issue
Section
AAAI Technical Track on Machine Learning I