Knowledge Refinery: Learning from Decoupled Label

Authors

  • Qianggang Ding Tsinghua University
  • Sifan Wu Tsinghua University
  • Tao Dai Tsinghua University PCL Research Center of Networks and Communications, Peng Cheng Laboratory
  • Hao Sun The Chinese University of Hong Kong
  • Jiadong Guo The Hong Kong University of Science and Technology International Digital Economy Academy
  • Zhang-Hua Fu The Chinese University of Hong Kong, Shenzhen Shenzhen Institute of Artificial Intelligence and Robotics for Society
  • Shutao Xia Tsinghua University PCL Research Center of Networks and Communications, Peng Cheng Laboratory

Keywords:

(Deep) Neural Network Algorithms, Classification and Regression

Abstract

Recently, a variety of regularization techniques have been widely applied in deep neural networks, which mainly focus on the regularization of weight parameters to encourage generalization effectively. Label regularization techniques are also proposed with the motivation of softening the labels while neglecting the relation of classes. Among them, the technique of knowledge distillation proposes to distill the soft label, which contains the knowledge of class relations. However, this technique needs to pre-train an extra cumbersome teacher model. In this paper, we propose a method called Knowledge Refinery (KR), which enables the neural network to learn the relation of classes on-the-fly without the teacher-student training strategy. We propose the definition of decoupled labels, which consist of the original hard label and the residual label. To exhibit the generalization of KR, we evaluate our method in both fields of computer vision and natural language processing. Our empirical results show consistent performance gains under all experimental settings.

Downloads

Published

2021-05-18

How to Cite

Ding, Q., Wu, S., Dai, T., Sun, H., Guo, J., Fu, Z.-H., & Xia, S. (2021). Knowledge Refinery: Learning from Decoupled Label. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7228-7235. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16888

Issue

Section

AAAI Technical Track on Machine Learning I