A Ground Truth Inference Model for Ordinal Crowd-Sourced Labels Using Hard Assignment Expectation Maximization

Abstract

We propose an iterative approach for inferring a ground truth value of an item from judgments collected form on-line workers. The method is specifically designed for cases in which the collected labels are ordinal. Our algorithm works by iteratively solving a hard-assignment EM model and later calculating one final expected value after the convergence of the EM procedure. This algorithm does not require any parameter tuning and can serve as turnkey algorithm for aggregating categorical and ordinal judgments.