What Makes A Good Story? Designing Composite Rewards for Visual Storytelling


  • Junjie Hu Carnegie Mellon University
  • Yu Cheng Microsoft Dynamics 365 AI Research
  • Zhe Gan Microsoft Dynamics 365 AI Research
  • Jingjing Liu Microsoft Dynamics 365 AI Research
  • Jianfeng Gao Microsoft Research
  • Graham Neubig Carnegie Mellon University




Previous storytelling approaches mostly focused on optimizing traditional metrics such as BLEU, ROUGE and CIDEr. In this paper, we re-examine this problem from a different angle, by looking deep into what defines a natural and topically-coherent story. To this end, we propose three assessment criteria: relevance, coherence and expressiveness, which we observe through empirical analysis could constitute a “high-quality” story to the human eye. We further propose a reinforcement learning framework, ReCo-RL, with reward functions designed to capture the essence of these quality criteria. Experiments on the Visual Storytelling Dataset (VIST) with both automatic and human evaluation demonstrate that our ReCo-RL model achieves better performance than state-of-the-art baselines on both traditional metrics and the proposed new criteria.




How to Cite

Hu, J., Cheng, Y., Gan, Z., Liu, J., Gao, J., & Neubig, G. (2020). What Makes A Good Story? Designing Composite Rewards for Visual Storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7969-7976. https://doi.org/10.1609/aaai.v34i05.6305



AAAI Technical Track: Natural Language Processing