Reinforced Multi-Teacher Selection for Knowledge Distillation

Authors

  • Fei Yuan University of Electronic Science and Technology of China
  • Linjun Shou Microsoft STCA NLP Group
  • Jian Pei School of Computing Science, Simon Fraser University
  • Wutao Lin Microsoft STCA NLP Group
  • Ming Gong Microsoft STCA NLP Group
  • Yan Fu University of Electronic Science and Technology of China
  • Daxin Jiang Microsoft STCA NLP Group

Keywords:

Applications

Abstract

In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models in production. As a popular method for model compression, knowledge distillation transfers knowledge from one or multiple large (teacher) models to a small (student) model. When multiple teacher models are available in distillation, the state-of-the-art methods assign a fixed weight to a teacher model in the whole distillation. Furthermore, most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled. We systematically develop a reinforced method to dynamically assign weights to teacher models for different training instances and optimize the performance of student model. Our extensive experimental results on several NLP tasks clearly verify the feasibility and effectiveness of our approach.

Downloads

Published

2021-05-18

How to Cite

Yuan, F., Shou, L., Pei, J., Lin, W., Gong, M., Fu, Y., & Jiang, D. (2021). Reinforced Multi-Teacher Selection for Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16), 14284-14291. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17680

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing III