Modeling Temporal Crowd Work Quality with Limited Supervision
DOI:
https://doi.org/10.1609/hcomp.v3i1.13227Keywords:
crowdsourcing, human computation, prediction, uncertainty-aware learning, time-series modelingAbstract
While recent work has shown that a worker’s performance can be more accurately modeled by temporal correlation in task performance, a fundamental challenge remains in the need for expert gold labels to evaluate a worker’s performance. To solve this problem, we explore two methods of utilizing limited gold labels, initial training and periodic updating. Furthermore, we present a novel way of learning a prediction model in the absence of gold labels with uncertaintyaware learning and soft-label updating. Our experiment with a real crowdsourcing dataset demonstrates that periodic updating tends to show better performance than initial training when the number of gold labels are very limited (< 25).