Unsupervised Deep Learning of Mid-Level Video Representation for Action Recognition


  • Jingyi Hou Beijing Institute of Technology
  • Xinxiao Wu Beijing Institute of Technology
  • Jin Chen Beijing Institute of Technology
  • Jiebo Luo University of Rochester
  • Yunde Jia Beijing Institute of Technology




Current deep learning methods for action recognition rely heavily on large scale labeled video datasets. Manually annotating video datasets is laborious and may introduce unexpected bias to train complex deep models for learning video representation. In this paper, we propose an unsupervised deep learning method which employs unlabeled local spatial-temporal volumes extracted from action videos to learn midlevel video representation for action recognition. Specifically, our method simultaneously discovers mid-level semantic concepts by discriminative clustering and optimizes local spatial-temporal features by two relatively small and simple deep neural networks. The clustering generates semantic visual concepts that guide the training of the deep networks, and the networks in turn guarantee the robustness of the semantic concepts. Experiments on the HMDB51 and the UCF101 datasets demonstrate the superiority of the proposed method, even over several supervised learning methods.




How to Cite

Hou, J., Wu, X., Chen, J., Luo, J., & Jia, Y. (2018). Unsupervised Deep Learning of Mid-Level Video Representation for Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.12300