Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering

Authors

  • Kaixin Ma Language Technologies Institute, School of Computer Science, Carnegie Mellon University
  • Filip Ilievski Information Sciences Institute, Viterbi School of Engineering, University of Southern California
  • Jonathan Francis Language Technologies Institute, School of Computer Science, Carnegie Mellon University Human-Machine Collaboration, Bosch Research Pittsburgh
  • Yonatan Bisk Language Technologies Institute, School of Computer Science, Carnegie Mellon University
  • Eric Nyberg Language Technologies Institute, School of Computer Science, Carnegie Mellon University
  • Alessandro Oltramari Human-Machine Collaboration, Bosch Research Pittsburgh

Keywords:

Question Answering, Common-Sense Reasoning, Unsupervised & Self-Supervised Learning, Neuro-Symbolic AI (NSAI)

Abstract

Recent developments in pre-trained neural language modeling have led to leaps in accuracy on common-sense question-answering benchmarks. However, there is increasing concern that models overfit to specific tasks, without learning to utilize external knowledge or perform general semantic reasoning. In contrast, zero-shot evaluations have shown promise as a more robust measure of a model’s general reasoning abilities. In this paper, we propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre-training models. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. Extending on prior work, we devise and compare four constrained distractor-sampling strategies. We provide empirical results across five commonsense question-answering tasks with data generated from five external knowledge resources. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks. In addition, both preserving the structure of the task as well as generating fair and informative questions help language models learn more effectively.

Downloads

Published

2021-05-18

How to Cite

Ma, K., Ilievski, F., Francis, J., Bisk, Y., Nyberg, E., & Oltramari, A. (2021). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15), 13507-13515. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17593

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing II