Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding


  • Zizhao Zhang Google
  • Han Zhang Google
  • Long Zhao Google
  • Ting Chen Google
  • Sercan Ö. Arik Google
  • Tomas Pfister Google



Computer Vision (CV)


Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8 times faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available




How to Cite

Zhang, Z., Zhang, H., Zhao, L., Chen, T., Arik, S. Ö., & Pfister, T. (2022). Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 3417-3425.



AAAI Technical Track on Computer Vision III