V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices


  • Damien Teney University of Adelaide
  • Peng Wang University of Wollongong
  • Jiewei Cao University of Adelaide
  • Lingqiao Liu University of Adelaide
  • Chunhua Shen University of Adelaide
  • Anton van den Hengel University of Adelaide




Advances in machine learning have generated increasing enthusiasm for tasks that require high-level reasoning on top of perceptual capabilities, particularly over visual data. Such tasks include, for example, image captioning, visual question answering, and visual navigation. Their evaluation is however hindered by task-specific confounding factors and dataset biases. In parallel, the existing benchmarks for abstract reasoning are limited to synthetic stimuli (e.g. images of simple shapes) and do not capture the challenges of real-world data. We propose a new large-scale benchmark to evaluates abstract reasoning over real visual data. The test involves visual questions that require operations fundamental to many high-level vision tasks, such as comparisons of counts and logical operations on complex visual properties. The benchmark measures a method's ability to infer high-level relationships and to generalise them over image-based concepts. We provide multiple training/test splits that require controlled levels of generalization. We evaluate a range of deep learning architectures, and find that existing models, including those popular for vision-and-language tasks, are unable to solve seemingly-simple instances. Models using relational networks fare better but leave substantial room for improvement.




How to Cite

Teney, D., Wang, P., Cao, J., Liu, L., Shen, C., & van den Hengel, A. (2020). V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 12071-12078. https://doi.org/10.1609/aaai.v34i07.6885



AAAI Technical Track: Vision