Leveraging Side Information to Improve Label Quality Control in Crowd-Sourcing


  • Yuan Jin Monash University
  • Mark Carman Monash University
  • Dongwoo Kim Australian National University
  • Lexing Xie Australian National University




Crowd-sourcing, Side Information, Probabilistic Modeling


We investigate the possibility of leveraging side information for improving quality control over crowd-sourced data. We extend the GLAD model, which governs the probability of correct labeling through a logistic function in which worker expertise counteracts item difficulty, by systematically encod- ing different types of side information, including worker in- formation drawn from demographics and personality traits, item information drawn from item genres and content, and contextual information drawn from worker responses and la- beling sessions. Modeling side information allows for better estimation of worker expertise and item difficulty in sparse data situations and accounts for worker biases, leading to bet- ter prediction of posterior true label probabilities. We demon- strate the efficacy of the proposed framework with overall improvements in both the true label prediction and the un- seen worker response prediction based on different combina- tions of the various types of side information across three new crowd-sourcing datasets. In addition, we show the framework exhibits potential of identifying salient side information fea- tures for predicting the correctness of responses without the need of knowing any true label information.




How to Cite

Jin, Y., Carman, M., Kim, D., & Xie, L. (2017). Leveraging Side Information to Improve Label Quality Control in Crowd-Sourcing. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 5(1), 79-88. https://doi.org/10.1609/hcomp.v5i1.13315