EM-Based Inference of True Labels Using Confidence Judgments


  • Satoshi Oyama Hokkaido University
  • Yukino Baba The University of Tokyo
  • Yuko Sakurai Kyushu University
  • Hisashi Kashima The University of Tokyo




crowdsourcing, human computation, quality control, confidence judgment, EM algorithm


We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.




How to Cite

Oyama, S., Baba, Y., Sakurai, Y., & Kashima, H. (2013). EM-Based Inference of True Labels Using Confidence Judgments. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 1(1), 58-59. https://doi.org/10.1609/hcomp.v1i1.13113