Cioppino: Multi-Tenant Crowd Management

Abstract

Embedding human computation in systems for data analysis improves the quality of the analysis, but can significantly impact the end-to-end cost and performance of the system. Recent work in crowdsourcing systems attempts to optimize for performance, but focuses on single applications running homogeneous tasks. In this work, we introduce Cioppino, a system that accounts for human factors that can affect performance when running multiple applications in parallel. Cioppino uses a queueing model to represent the pool of workers, and leverages techniques for autoscaling used in cloud computing to adaptively adjust the pool size. Its model also accounts for worker abandonment, and automatically shifts workers between applications to improve performance and match workers with tasks they enjoy most. Our evaluation of Cioppino in simulation on traces extracted from a realtime crowd system running on Amazon’s Mechanical Turk demonstrates a 19X reduction in cost, a 20% increase in throughput, and a 2X increase in worker preference for assigned tasks as compared to state-of-the-art crowd management strategies.