To Re(label), or Not To Re(label)


  • Christopher Lin University of Washington
  • . Mausam Indian Institute of Technology, Delhi
  • Daniel Weld University of Washington


relabeling, crowdsourcing, machine learning


One of the most popular uses of crowdsourcing is to provide training data for supervised machine learning algorithms. Since human annotators often make errors, requesters commonly ask multiple workers to label each example.  But is this strategy always the most cost effective use of crowdsourced workers? We argue "No" --- often classifiers can achieve higher accuracies when trained with noisy "unilabeled" data. However, in some cases relabeling is extremely important.  We discuss three factors that may make relabeling an effective strategy: classifier expressiveness, worker accuracy, and budget.




How to Cite

Lin, C., Mausam, ., & Weld, D. (2014). To Re(label), or Not To Re(label). Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2(1). Retrieved from