Crowdsourcing Quality Control for Item Ordering Tasks


  • Toshiko Matsui The University of Tokyo
  • Yukino Baba The University of Tokyo
  • Toshihiro Kamishima National Institute of Advanced Industrial Science and Technology
  • Hisashi Kashima The University of Tokyo and JST PRESTO



One of the biggest challenges in crowdsourcing is quality control which is to expect high quality results from crowd workers who are not necessarily very capable nor motivated.In this paper, we consider item ordering questions, where workersare asked to arrange multiple items in the correct order. We propose a probabilistic generative model of crowd answers by extending a distance-based order model to incorporate worker ability, and give an efficient estimation algorithm.




How to Cite

Matsui, T., Baba, Y., Kamishima, T., & Kashima, H. (2013). Crowdsourcing Quality Control for Item Ordering Tasks. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 1(1), 52-53.