Activity Image-to-Video Retrieval by Disentangling Appearance and Motion

Authors

  • Liu Liu Shanghai Jiao Tong University
  • Jiangtong Li Shanghai Jiao Tong University
  • Li Niu Shanghai Jiao Tong University
  • Ruicong Xu MEITUAN
  • Liqing Zhang Shanghai Jiao Tong University

Keywords:

Image and Video Retrieval

Abstract

With the rapid emergence of video data, image-to-video retrieval has attracted much attention. There are two types of image-to-video retrieval: instance-based and activity-based. The former task aims to retrieve videos containing the same main objects as the query image, while the latter focuses on finding the similar activity. Since dynamic information plays a significant role in the video, we pay attention to the latter task to explore the motion relation between images and videos. In this paper, we propose a Motion-assisted Activity Proposal-based Image-to-Video Retrieval (MAP-IVR) approach to disentangle the video features into motion features and appearance features and obtain appearance features from the images. Then, we perform image-to-video translation to improve the disentanglement quality. The retrieval is performed in both appearance and video feature spaces. Extensive experiments demonstrate that our MAP-IVR approach remarkably outperforms the state-of-the-art approaches on two benchmark activity-based video datasets.

Downloads

Published

2021-05-18

How to Cite

Liu, L. ., Li, J., Niu, L., Xu, R., & Zhang, L. (2021). Activity Image-to-Video Retrieval by Disentangling Appearance and Motion. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 2145-2153. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16312

Issue

Section

AAAI Technical Track on Computer Vision II