A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing


  • Ting-Hao Huang Carnegie Mellon University
  • Jeffrey Bigham Carnegie Mellon University




crowdsourcing, crowd-powered system


A number of interactive crowd-powered systems have been developed to solve difficult problems out of reach for automated solutions. To work interactively, such systems need access to on-demand labor. To meet this demand, workers can be (i) recruited when needed directly from the crowd marketplace, or (ii) recruited in advance and asked to wait in a retainer pool until they are needed. Most of the evaluations of these systems have been over a short time period, even though we know that marketplaces change and adapt over time. In this paper, we present the results of a 10-month deployment of a crowd-powered system that uses a hybrid approach to fast recruitment of workers that we call Ignition. We describe the Ignition approach and the observed times required to recruit workers from the marketplace and retainer over this long period of time. Our results demonstrate that it is possible to recruit workers with low latency even over long periods, and suggest a number of opportunities for future work for recruitment strategies and modeling that may further improve on-demand recruitment for deployed systems.




How to Cite

Huang, T.-H., & Bigham, J. (2017). A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 5(1), 61-70. https://doi.org/10.1609/hcomp.v5i1.13318