A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis

Authors

  • Shai Gretz IBM Research AI
  • Roni Friedman IBM Research AI
  • Edo Cohen-Karlik IBM Research AI
  • Assaf Toledo IBM Research AI
  • Dan Lahav IBM Research AI
  • Ranit Aharonov IBM Research AI
  • Noam Slonim IBM Research AI

DOI:

https://doi.org/10.1609/aaai.v34i05.6285

Abstract

Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.

Downloads

Published

2020-04-03

How to Cite

Gretz, S., Friedman, R., Cohen-Karlik, E., Toledo, A., Lahav, D., Aharonov, R., & Slonim, N. (2020). A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7805-7813. https://doi.org/10.1609/aaai.v34i05.6285

Issue

Section

AAAI Technical Track: Natural Language Processing