Dual Compositional Learning in Interactive Image Retrieval


  • Jongseok Kim Seoul National University
  • Youngjae Yu Seoul National University RippleAI
  • Hoeseong Kim Seoul National University
  • Gunhee Kim Seoul National University RippleAI




Image and Video Retrieval


We present an approach named Dual Composition Network (DCNet) for interactive image retrieval that searches for the best target image for a natural language query and a reference image. To accomplish this task, existing methods have focused on learning a composite representation of the reference image and the text query to be as close to the embedding of the target image as possible. We refer this approach as Composition Network. In this work, we propose to close the loop with Correction Network that models the difference between the reference and target image in the embedding space and matches it with the embedding of the text query. That is, we consider two cyclic directional mappings for triplets of (reference image, text query, target image) by using both Composition Network and Correction Network. We also propose a joint training loss that can further improve the robustness of multimodal representation learning. We evaluate the proposed model on three benchmark datasets for multimodal retrieval: Fashion-IQ, Shoes, and Fashion200K. Our experiments show that our DCNet achieves new state-of-the-art performance on all three datasets, and the addition of Correction Network consistently improves multiple existing methods that are solely based on Composition Network. Moreover, an ensemble of our model won the first place in Fashion-IQ 2020 challenge held in a CVPR 2020 workshop.




How to Cite

Kim, J., Yu, Y., Kim, H., & Kim, G. (2021). Dual Compositional Learning in Interactive Image Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1771-1779. https://doi.org/10.1609/aaai.v35i2.16271



AAAI Technical Track on Computer Vision I