Interactive Dual Generative Adversarial Networks for Image Captioning

Authors

  • Junhao Liu Chinese Academy of Sciences
  • Kai Wang Chinese Academy of Sciences
  • Chunpu Xu Huazhong University of Science and Technology
  • Zhou Zhao Zhejiang University
  • Ruifeng Xu Harbin Institute of Technology (Shenzhen)
  • Ying Shen Peking University Shenzhen Graduate School
  • Min Yang Chinese Academy of Sciences

DOI:

https://doi.org/10.1609/aaai.v34i07.6826

Abstract

Image captioning is usually built on either generation-based or retrieval-based approaches. Both ways have certain strengths but suffer from their own limitations. In this paper, we propose an Interactive Dual Generative Adversarial Network (IDGAN) for image captioning, which mutually combines the retrieval-based and generation-based methods to learn a better image captioning ensemble. IDGAN consists of two generators and two discriminators, where the generation- and retrieval-based generators mutually benefit from each other's complementary targets that are learned from two dual adversarial discriminators. Specifically, the generation- and retrieval-based generators provide improved synthetic and retrieved candidate captions with informative feedback signals from the two respective discriminators that are trained to distinguish the generated captions from the true captions and assign top rankings to true captions respectively, thus featuring the merits of both retrieval-based and generation-based approaches. Extensive experiments on MSCOCO dataset demonstrate that the proposed IDGAN model significantly outperforms the compared methods for image captioning.

Downloads

Published

2020-04-03

How to Cite

Liu, J., Wang, K., Xu, C., Zhao, Z., Xu, R., Shen, Y., & Yang, M. (2020). Interactive Dual Generative Adversarial Networks for Image Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 11588-11595. https://doi.org/10.1609/aaai.v34i07.6826

Issue

Section

AAAI Technical Track: Vision