Learning Multimodal Word Representation via Dynamic Fusion Methods

Authors

  • Shaonan Wang Institute of Automation, Chinese Academy of Sciences
  • Jiajun Zhang Institute of Automation, Chinese Academy of Sciences
  • Chengqing Zong Institute of Automation, Chinese Academy of Sciences

Keywords:

Multimodal word representations, dynamic fusion, word associations

Abstract

Multimodal models have been proven to outperform text-based models on learning semantic word representations. Almost all previous multimodal models typically treat the representations from different modalities equally. However, it is obvious that information from different modalities contributes differently to the meaning of words. This motivates us to build a multimodal model that can dynamically fuse the semantic representations from different modalities according to different types of words. To that end, we propose three novel dynamic fusion methods to assign importance weights to each modality, in which weights are learned under the weak supervision of word association pairs. The extensive experiments have demonstrated that the proposed methods outperform strong unimodal baselines and state-of-the-art multimodal models.

Downloads

Published

2018-04-26

How to Cite

Wang, S., Zhang, J., & Zong, C. (2018). Learning Multimodal Word Representation via Dynamic Fusion Methods. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/12031