Overcoming Language Priors in VQA via Decomposed Linguistic Representations

Authors

  • Chenchen Jing Beijing Institute of Technology
  • Yuwei Wu Beijing Institute of Technology
  • Xiaoxun Zhang Alibaba Group
  • Yunde Jia Beijing Institute of Technology
  • Qi Wu University of Adelaide

DOI:

https://doi.org/10.1609/aaai.v34i07.6776

Abstract

Most existing Visual Question Answering (VQA) models overly rely on language priors between questions and answers. In this paper, we present a novel method of language attention-based VQA that learns decomposed linguistic representations of questions and utilizes the representations to infer answers for overcoming language priors. We introduce a modular language attention mechanism to parse a question into three phrase representations: type representation, object representation, and concept representation. We use the type representation to identify the question type and the possible answer set (yes/no or specific concepts such as colors or numbers), and the object representation to focus on the relevant region of an image. The concept representation is verified with the attended region to infer the final answer. The proposed method decouples the language-based concept discovery and vision-based concept verification in the process of answer inference to prevent language priors from dominating the answering process. Experiments on the VQA-CP dataset demonstrate the effectiveness of our method.

Downloads

Published

2020-04-03

How to Cite

Jing, C., Wu, Y., Zhang, X., Jia, Y., & Wu, Q. (2020). Overcoming Language Priors in VQA via Decomposed Linguistic Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 11181-11188. https://doi.org/10.1609/aaai.v34i07.6776

Issue

Section

AAAI Technical Track: Vision