Learning to Answer Questions from Image Using Convolutional Neural Network

Authors

  • Lin Ma Noah’s Ark Lab, Huawei Technologies
  • Zhengdong Lu Noah’s Ark Lab, Huawei Technologies
  • Hang Li Noah’s Ark Lab, Huawei Technologies

DOI:

https://doi.org/10.1609/aaai.v30i1.10442

Keywords:

image and language, image question answering, CNN

Abstract

In this paper, we propose to employ the convolutional neural network (CNN) for the image question answering (QA) task. Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: one image CNN to encode the image content, one sentence CNN to compose the words of the question, and one multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on the DAQUAR and COCO-QA datasets, which are two benchmark datasets for image QA, with the performances significantly outperforming the state-of-the-art.

Downloads

Published

2016-03-05

How to Cite

Ma, L., Lu, Z., & Li, H. (2016). Learning to Answer Questions from Image Using Convolutional Neural Network. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10442