Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning


  • Jingwen Chen Sun Yat-sen University
  • Yingwei Pan JD AI Research
  • Yehao Li Sun Yat-sen University
  • Ting Yao JD AI Research
  • Hongyang Chao Sun Yat-sen University
  • Tao Mei JD AI Research




It is well believed that video captioning is a fundamental but challenging task in both computer vision and artificial intelligence fields. The prevalent approach is to map an input video to a variable-length output sentence in a sequence to sequence manner via Recurrent Neural Network (RNN). Nevertheless, the training of RNN still suffers to some degree from vanishing/exploding gradient problem, making the optimization difficult. Moreover, the inherently recurrent dependency in RNN prevents parallelization within a sequence during training and therefore limits the computations. In this paper, we present a novel design — Temporal Deformable Convolutional Encoder-Decoder Networks (dubbed as TDConvED) that fully employ convolutions in both encoder and decoder networks for video captioning. Technically, we exploit convolutional block structures that compute intermediate states of a fixed number of inputs and stack several blocks to capture long-term relationships. The structure in encoder is further equipped with temporal deformable convolution to enable free-form deformation of temporal sampling. Our model also capitalizes on temporal attention mechanism for sentence generation. Extensive experiments are conducted on both MSVD and MSR-VTT video captioning datasets, and superior results are reported when comparing to conventional RNN-based encoder-decoder techniques. More remarkably, TDConvED increases CIDEr-D performance from 58.8% to 67.2% on MSVD.




How to Cite

Chen, J., Pan, Y., Li, Y., Yao, T., Chao, H., & Mei, T. (2019). Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 8167-8174. https://doi.org/10.1609/aaai.v33i01.33018167



AAAI Technical Track: Vision