DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval

Authors

  • Xiangpeng Yang University of Technology, Sydney
  • Linchao Zhu Zhejiang University
  • Xiaohan Wang Zhejiang University
  • Yi Yang Zhejiang University

DOI:

https://doi.org/10.1609/aaai.v38i7.28475

Keywords:

CV: Language and Vision, CV: Image and Video Retrieval

Abstract

Text-video retrieval is a critical multi-modal task to find the most relevant video for a text query. Although pretrained models like CLIP have demonstrated impressive potential in this area, the rising cost of fully finetuning these models due to increasing model size continues to pose a problem. To address this challenge, prompt tuning has emerged as an alternative. However, existing works still face two problems when adapting pretrained image-text models to downstream video-text tasks: (1) The visual encoder could only encode frame-level features and failed to extract global-level general video information. (2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap. To this end, we propose DGL, a cross-modal Dynamic prompt tuning method with Global-Local video attention. In contrast to previous prompt tuning methods, we employ the shared latent space to generate local-level text and frame prompts that encourage inter-modal interaction. Furthermore, we propose modeling video in a global-local attention mechanism to capture global video information from the perspective of prompt tuning. Extensive experiments reveal that when only 0.67% parameters are tuned, our cross-modal prompt tuning strategy DGL outperforms or is comparable to fully finetuning methods on MSR-VTT, VATEX, LSMDC, and ActivityNet datasets. Code will be available at https://github.com/knightyxp/DGL.

Published

2024-03-24

How to Cite

Yang, X., Zhu, L., Wang, X., & Yang, Y. (2024). DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7), 6540-6548. https://doi.org/10.1609/aaai.v38i7.28475

Issue

Section

AAAI Technical Track on Computer Vision VI