Utilizing Crowdsourced Asynchronous Chat for Efficient Collection of Dialogue Dataset

Authors

  • Kazushi Ikeda KDDI Research, Inc.
  • Keiichiro Hoashi KDDI Research, Inc.

DOI:

https://doi.org/10.1609/hcomp.v6i1.13321

Keywords:

conversational agent, dialogue data collection, efficiency, crowdsourcing

Abstract

In this paper, we design a crowd-powered system to efficiently collect data for training dialogue systems. Conventional systems assign dialogue roles to a pair of crowd workers, and record their interaction on an online chat. In this framework, the pair is required to work simultaneously, and one worker must wait for the other when he/she is writing a message, which decreases work efficiency. Our proposed system allows multiple workers to create dialogues in an asynchronous manner, which relieves workers from time restrictions. We have conducted an experiment using our system on a crowdsourcing platform to evaluate the efficiency and the quality of dialogue collection. Results show that our system can reduce the necessary time to input a message by 68% while maintaining quality.

Downloads

Published

2018-06-15

How to Cite

Ikeda, K., & Hoashi, K. (2018). Utilizing Crowdsourced Asynchronous Chat for Efficient Collection of Dialogue Dataset. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6(1), 60-69. https://doi.org/10.1609/hcomp.v6i1.13321