Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation
Keywords:Speech & Natural Language Processing (SNLP)
AbstractThe goal of a summary is to concisely state the most important information in a document. With this principle in mind, we introduce new reference-free summary evaluation metrics that use a pretrained language model to estimate the information content shared between a document and its summary. These metrics are a modern take on the Shannon Game, a method for summary quality scoring proposed decades ago, where we replace human annotators with language models. We also view these metrics as an extension of BLANC, a recently proposed approach to summary quality measurement based on the performance of a language model with and without the help of a summary. Using transformer based language models, we empirically verify that our metrics achieve state-of-the-art correlation with human judgement of the summary quality dimensions of both coherence and relevance, as well as competitive correlation with human judgement of consistency and fluency.
How to Cite
Egan, N., Vasilyev, O., & Bohannon, J. (2022). Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10599-10607. https://doi.org/10.1609/aaai.v36i10.21304
AAAI Technical Track on Speech and Natural Language Processing