Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding


  • Zhenzhi Wang Nanjing University
  • Limin Wang Nanjing University
  • Tao Wu Nanjing University
  • Tianhao Li Nanjing University
  • Gangshan Wu Nanjing University



Computer Vision (CV), Speech & Natural Language Processing (SNLP)


Temporal grounding aims to localize a video moment which is semantically aligned with a given natural language query. Existing methods typically apply a detection or regression pipeline on the fused representation with the research focus on designing complicated prediction heads or fusion strategies. Instead, from a perspective on temporal grounding as a metric-learning problem, we present a Mutual Matching Network (MMN), to directly model the similarity between language queries and video moments in a joint embedding space. This new metric-learning framework enables fully exploiting negative samples from two new aspects: constructing negative cross-modal pairs in a mutual matching scheme and mining negative pairs across different videos. These new negative samples could enhance the joint representation learning of two modalities via cross-modal mutual matching to maximize their mutual information. Experiments show that our MMN achieves highly competitive performance compared with the state-of-the-art methods on four video grounding benchmarks. Based on MMN, we present a winner solution for the HC-STVG challenge of the 3rd PIC workshop. This suggests that metric learning is still a promising method for temporal grounding via capturing the essential cross-modal correlation in a joint embedding space. Code is available at




How to Cite

Wang, Z., Wang, L., Wu, T., Li, T., & Wu, G. (2022). Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 2613-2623.



AAAI Technical Track on Computer Vision III