Language Modelling via Learning to Rank
DOI:
https://doi.org/10.1609/aaai.v36i10.21308Keywords:
Speech & Natural Language Processing (SNLP), Machine Learning (ML)Abstract
We consider language modelling (LM) as a multi-label structured prediction task by re-framing training from solely predicting a single ground-truth word to ranking a set of words which could continue a given context. To avoid annotating top-k ranks, we generate them using pre-trained LMs: GPT-2, BERT, and Born-Again models. This leads to a rank-based form of knowledge distillation (KD). We also develop a method using N-grams to create a non-probabilistic teacher which generates the ranks without the need of a pre-trained LM. We confirm the hypotheses: that we can treat LMing as a ranking task and that we can do so without the use of a pre-trained LM. We show that rank-based KD generally gives a modest improvement to perplexity (PPL) -- though often with statistical significance -- when compared to Kullback–Leibler-based KD. Surprisingly, given the naivety of the method, the N-grams act as competitive teachers and achieve similar performance as using either BERT or a Born-Again model teachers. Unsurprisingly, GPT-2 always acts as the best teacher. Using it and a Transformer-XL student on Wiki-02, rank-based KD reduces a cross-entropy baseline from 65.27 to 55.94 and against a KL-based KD of 56.70.Downloads
Published
2022-06-28
How to Cite
Frydenlund, A., Singh, G., & Rudzicz, F. (2022). Language Modelling via Learning to Rank. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10636-10644. https://doi.org/10.1609/aaai.v36i10.21308
Issue
Section
AAAI Technical Track on Speech and Natural Language Processing