Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup


  • Sijia Liu Amazon Alexa AI
  • Patrick Lange Amazon Alexa AI
  • Behnam Hedayatnia Amazon Alexa AI
  • Alexandros Papangelis Amazon Alexa AI
  • Di Jin Amazon Alexa AI
  • Andrew Wirth Amazon Alexa AI
  • Yang Liu Amazon Alexa AI
  • Dilek Hakkani-Tur Amazon Alexa AI



SNLP: Conversational AI/Dialogue Systems, SNLP: Interpretability & Analysis of NLP Models


Evaluating open-domain conversation models has been an open challenge due to the open-ended nature of conversations. In addition to static evaluations, recent work has started to explore a variety of per-turn and per-dialog interactive evaluation mechanisms and provide advice on the best setup. In this work, we adopt the interactive evaluation framework and further apply to multiple models with a focus on per-turn evaluation techniques. Apart from the widely used setting where participants select the best response among different candidates at each turn, one more novel per-turn evaluation setting is adopted, where participants can select all appropriate responses with different fallback strategies to continue the conversation when no response is selected. We evaluate these settings based on sensitivity and consistency using four GPT2-based models that differ in model sizes or fine-tuning data. To better generalize to any model groups with no prior assumptions on their rankings and control evaluation costs for all setups, we also propose a methodology to estimate the required sample size given a minimum performance gap of interest before running most experiments. Our comprehensive human evaluation results shed light on how to conduct credible human evaluations of open domain dialog systems using the interactive setup, and suggest additional future directions.




How to Cite

Liu, S., Lange, P., Hedayatnia, B., Papangelis, A., Jin, D., Wirth, A., Liu, Y., & Hakkani-Tur, D. (2023). Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13264-13272.



AAAI Technical Track on Speech & Natural Language Processing