Crowdsourcing Quality Control for Item Ordering Tasks

Authors

  • Toshiko Matsui The University of Tokyo
  • Yukino Baba The University of Tokyo
  • Toshihiro Kamishima National Institute of Advanced Industrial Science and Technology
  • Hisashi Kashima The University of Tokyo and JST PRESTO

DOI:

https://doi.org/10.1609/hcomp.v1i1.13106

Abstract

One of the biggest challenges in crowdsourcing is quality control which is to expect high quality results from crowd workers who are not necessarily very capable nor motivated.In this paper, we consider item ordering questions, where workersare asked to arrange multiple items in the correct order. We propose a probabilistic generative model of crowd answers by extending a distance-based order model to incorporate worker ability, and give an efficient estimation algorithm.

Downloads

Published

2013-11-03

How to Cite

Matsui, T., Baba, Y., Kamishima, T., & Kashima, H. (2013). Crowdsourcing Quality Control for Item Ordering Tasks. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 1(1), 52-53. https://doi.org/10.1609/hcomp.v1i1.13106

Issue

Vol. 1 (2013): First AAAI Conference on Human Computation and Crowdsourcing

Section

Works in Progress