UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

Authors

  • Nicholas Lourie Allen Institute for AI
  • Ronan Le Bras Allen Institute for AI
  • Chandra Bhagavatula Allen Institute for AI
  • Yejin Choi Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for AI

DOI:

https://doi.org/10.1609/aaai.v35i15.17590

Keywords:

Language Models, Question Answering

Abstract

Commonsense AI has long been seen as a near impossible goal---until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose two new ways to evaluate commonsense models, emphasizing their generality on new tasks and building on diverse, recently introduced benchmarks. First, we propose a new multitask benchmark, Rainbow, to promote research on commonsense models that generalize well over multiple tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how the choice of source datasets, pretrained language models, and transfer learning methods impacts performance and data efficiency. We perform extensive experiments---over 200 experiments encompassing 4800 models---and report multiple valuable and sometimes surprising findings, e.g., that transfer almost always leads to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well with each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit more from transfer than smaller ones. Last but not least, we introduce a new universal commonsense reasoning model, UNICORN, that establishes new state-of-the-art performance across 8 popular commonsense benchmarks, aNLI (87.3%), CosmosQA (91.8%), HellaSWAG (93.9%), PIQA (90.1%), SocialIQa (83.2%), WinoGrande (86.6%), CycIC (94.0%) and CommonsenseQA (79.3%).

Downloads

Published

2021-05-18

How to Cite

Lourie, N., Le Bras, R., Bhagavatula, C., & Choi, Y. (2021). UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15), 13480-13488. https://doi.org/10.1609/aaai.v35i15.17590

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing II