Less Is More: Pay Less Attention in Vision Transformers


  • Zizheng Pan Monash University
  • Bohan Zhuang Monash University
  • Haoyu He Monash University
  • Jing Liu Monash University
  • Jianfei Cai Monash University




Computer Vision (CV)


Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) in computer vision. However, Transformer training and inference in previous works can be prohibitively expensive due to the quadratic complexity of self-attention over a long sequence of representations, especially for high-resolution dense prediction tasks. To this end, we present a novel Less attention vIsion Transformer (LIT), building upon the fact that the early self-attention layers in Transformers still focus on local patterns and bring minor benefits in recent hierarchical vision Transformers. Specifically, we propose a hierarchical Transformer where we use pure multi-layer perceptrons (MLPs) to encode rich local patterns in the early stages while applying self-attention modules to capture longer dependencies in deeper layers. Moreover, we further propose a learned deformable token merging module to adaptively fuse informative patches in a non-uniform manner. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation, serving as a strong backbone for many vision tasks. Code is available at https://github.com/zip-group/LIT.




How to Cite

Pan, Z., Zhuang, B., He, H., Liu, J., & Cai, J. (2022). Less Is More: Pay Less Attention in Vision Transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), 2035-2043. https://doi.org/10.1609/aaai.v36i2.20099



AAAI Technical Track on Computer Vision II