Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation
DOI:
https://doi.org/10.1609/aaai.v36i10.21304Keywords:
Speech & Natural Language Processing (SNLP)Abstract
The goal of a summary is to concisely state the most important information in a document. With this principle in mind, we introduce new reference-free summary evaluation metrics that use a pretrained language model to estimate the information content shared between a document and its summary. These metrics are a modern take on the Shannon Game, a method for summary quality scoring proposed decades ago, where we replace human annotators with language models. We also view these metrics as an extension of BLANC, a recently proposed approach to summary quality measurement based on the performance of a language model with and without the help of a summary. Using transformer based language models, we empirically verify that our metrics achieve state-of-the-art correlation with human judgement of the summary quality dimensions of both coherence and relevance, as well as competitive correlation with human judgement of consistency and fluency.Downloads
Published
2022-06-28
How to Cite
Egan, N., Vasilyev, O., & Bohannon, J. (2022). Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10599-10607. https://doi.org/10.1609/aaai.v36i10.21304
Issue
Section
AAAI Technical Track on Speech and Natural Language Processing