TY - JOUR AU - Zhang, Zizhao AU - Zhang, Han AU - Zhao, Long AU - Chen, Ting AU - Arik, Sercan Ö. AU - Pfister, Tomas PY - 2022/06/28 Y2 - 2024/03/29 TI - Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 36 IS - 3 SE - AAAI Technical Track on Computer Vision III DO - 10.1609/aaai.v36i3.20252 UR - https://ojs.aaai.org/index.php/AAAI/article/view/20252 SP - 3417-3425 AB - Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8 times faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available https://github.com/google-research/nested-transformer. ER -