Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition


  • Zhan Chen Peking University Shenzhen Graduate School
  • Sicheng Li Zhejiang University
  • Bing Yang Peking University Shenzhen Graduate School
  • Qinghan Li Syracuse University
  • Hong Liu Peking University Shenzhen Graduate School



Video Understanding & Activity Analysis


Graph convolutional networks have been widely used for skeleton-based action recognition due to their excellent modeling ability of non-Euclidean data. As the graph convolution is a local operation, it can only utilize the short-range joint dependencies and short-term trajectory but fails to directly model the distant joints relations and long-range temporal information that are vital to distinguishing various actions. To solve this problem, we present a multi-scale spatial graph convolution (MS-GC) module and a multi-scale temporal graph convolution (MT-GC) module to enrich the receptive field of the model in spatial and temporal dimensions. Concretely, the MS-GC and MT-GC modules decompose the corresponding local graph convolution into a set of sub-graph convolution, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-graph convolutions, and each node could complete multiple spatial and temporal aggregations with its neighborhoods. The final equivalent receptive field is accordingly enlarged, which is capable of capturing both short- and long-range dependencies in spatial and temporal domains. By coupling these two modules as a basic block, we further propose a multi-scale spatial temporal graph convolutional network (MST-GCN), which stacks multiple blocks to learn effective motion representations for action recognition. The proposed MST-GCN achieves remarkable performance on three challenging benchmark datasets, NTU RGB+D, NTU-120 RGB+D and Kinetics-Skeleton, for skeleton-based action recognition.




How to Cite

Chen, Z., Li, S., Yang, B., Li, Q., & Liu, H. (2021). Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1113-1122.



AAAI Technical Track on Computer Vision I