Progression in a Language Annotation Game with a Purpose

Authors

  • Chris Madge Queen Mary University of London
  • Juntao Yu Queen Mary University of London
  • Jon Chamberlain University of Essex
  • Udo Kruschwitz University of Regensburg
  • Silviu Paun Queen Mary University of London
  • Massimo Poesio Queen Mary University of London

Abstract

Within traditional games design, incorporating progressive difficulty is considered of fundamental importance. But despite the widespread intuition that progression could have clear benefits in Games-With-A-Purpose (GWAPs)–e.g., for training non-expert annotators to produce more complex judgements– progression is not in fact a prominent feature of GWAPs; and there is even less evidence on its effects. In this work we present an approach to progression in GWAPs that generalizes to different annotation tasks with minimal, if any, dependency on gold annotated data. Using this method we observe a statistically significant increase in accuracy over randomly showing items to annotators.

Downloads

Published

2019-10-28

How to Cite

Madge, C., Yu, J., Chamberlain, J., Kruschwitz, U., Paun, S., & Poesio, M. (2019). Progression in a Language Annotation Game with a Purpose. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 77-85. Retrieved from https://ojs.aaai.org/index.php/HCOMP/article/view/5276