Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

Authors

  • Steven Y. Feng Language Technologies Institute, Carnegie Mellon University
  • Kevin Lu University of Waterloo
  • Zhuofu Tao University of California, Los Angeles
  • Malihe Alikhani School of Computing and Information, University of Pittsburgh
  • Teruko Mitamura Language Technologies Institute, Carnegie Mellon University
  • Eduard Hovy Language Technologies Institute, Carnegie Mellon University
  • Varun Gangal Language Technologies Institute, Carnegie Mellon University

DOI:

https://doi.org/10.1609/aaai.v36i10.21306

Keywords:

Speech & Natural Language Processing (SNLP), Machine Learning (ML)

Abstract

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.
AAAI-22 Proceedings Cover

Downloads

Published

2022-06-28

How to Cite

Feng, S. Y., Lu, K., Tao, Z., Alikhani, M., Mitamura, T., Hovy, E., & Gangal, V. (2022). Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10618-10626. https://doi.org/10.1609/aaai.v36i10.21306

Issue

Vol. 36 No. 10: AAAI-22 Technical Tracks 10

Section

AAAI Technical Track on Speech and Natural Language Processing