Visual Definition Modeling: Challenging Vision & Language Models to Define Words and Objects

Authors

  • Bianca Scarlini Sapienza University of Rome
  • Tommaso Pasini Computer Science Department, University of Copenhagen
  • Roberto Navigli Sapienza University of Rome

DOI:

https://doi.org/10.1609/aaai.v36i10.21377

Keywords:

Speech & Natural Language Processing (SNLP)

Abstract

Architectures that model language and vision together havereceived much attention in recent years. Nonetheless, most tasks in this field focus on end-to-end applications without providing insights on whether it is the underlying semantics of visual objects or words that is captured. In this paper we draw on the established Definition Modeling paradigm and enhance it by grounding, for the first time, textual definitions to visual representations. We name this new task Visual Definition Modeling and put forward DEMETER and DIONYSUS, two benchmarks where, given an image as context, models have to generate a textual definition for a target being either i) a word that describes the image, or ii) an object patch therein. To measure the difficulty of our tasks we finetuned six different baselines and analyzed their performances, which show that a text-only encoder-decoder model is more effective than models pretrained for handling inputs of both modalities concurrently. This demonstrates the complexity of our benchmarks and encourages more research on text generation conditioned on multimodal inputs. The datasets for both benchmarks are available at https://github.com/SapienzaNLP/visual-definition-modeling as well as the code to reproduce our models.

Downloads

Published

2022-06-28

How to Cite

Scarlini, B., Pasini, T., & Navigli, R. (2022). Visual Definition Modeling: Challenging Vision & Language Models to Define Words and Objects. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11267-11275. https://doi.org/10.1609/aaai.v36i10.21377

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing