Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff

Authors

  • Jiawei Liu Wuhan University
  • Zhe Gao Alibaba Group
  • Yangyang Kang Alibaba Group
  • Zhuoren Jiang Zhejiang University
  • Guoxiu He Wuhan University
  • Changlong Sun Alibaba Group Zhejiang University
  • Xiaozhong Liu Indiana University Bloomington
  • Wei Lu Wuhan University

Keywords:

Applications, Conversational AI/Dialog Systems

Abstract

Is chatbot able to completely replace the human agent? The short answer could be – ``it depends...''. For some challenging cases, e.g., dialogue's topical spectrum spreads beyond the training corpus coverage, the chatbot may malfunction and return unsatisfied utterances. This problem can be addressed by introducing the Machine-Human Chatting Handoff (MHCH) which enables human-algorithm collaboration. To detect the normal/transferable utterances, we propose a Difficulty-Assisted Matching Inference (DAMI) network, utilizing difficulty-assisted encoding to enhance the representations of utterances. Moreover, a matching inference mechanism is introduced to capture the contextual matching features. A new evaluation metric, Golden Transfer within Tolerance (GT-T), is proposed to assess the performance by considering the tolerance property of the MHCH. To provide insights into the task and validate the proposed model, we collect two new datasets. Extensive experimental results are presented and contrasted against a series of baseline models to demonstrate the efficacy of our model on MHCH.

Downloads

Published

2021-05-18

How to Cite

Liu, J., Gao, Z., Kang, Y., Jiang, Z., He, G., Sun, C., Liu, X., & Lu, W. (2021). Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7), 5841-5849. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16731

Issue

Section

AAAI Technical Track on Human-Computation and Crowd Sourcing