Proposal-Free Video Grounding with Contextual Pyramid Network

Authors

  • Kun Li Hefei University of Technology
  • Dan Guo Hefei University of Technology
  • Meng Wang Hefei University of Technology

Keywords:

Language and Vision, Video Understanding & Activity Analysis

Abstract

The challenge of video grounding - localizing activities in an untrimmed video via a natural language query - is to tackle the semantics of vision and language consistently along the temporal dimension. Most existing proposal-based methods are trapped by computational cost with extensive candidate proposals. In this paper, we propose a novel proposal-free framework named Contextual Pyramid Network (CPNet) to investigate multi-scale temporal correlation in the video. Specifically, we propose a pyramid network to extract 2D contextual correlation maps at different temporal scales (T*T, T/2*T/2, T/4*T/4), where the 2D correlation map (past to current & future to current) is designed to model all the relations of any two moments in the video. In other words, CPNet progressively replenishes the temporal contexts and refines the location of queried activity by enlarging the temporal receptive fields. Finally, we implement a temporal self-attentive regression (i.e., proposal-free regression) to predict the activity boundary from the above hierarchical context-aware 2D correlation maps. Extensive experiments on ActivityNet Captions, Charades-STA, and TACoS datasets demonstrate that our approach outperforms state-of-the-art methods.

Downloads

Published

2021-05-18

How to Cite

Li, K., Guo, D., & Wang, M. (2021). Proposal-Free Video Grounding with Contextual Pyramid Network. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 1902-1910. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16285

Issue

Section

AAAI Technical Track on Computer Vision II