Knowledge Distillation with Adversarial Samples Supporting Decision Boundary


  • Byeongho Heo Seoul National University
  • Minsik Lee Hanyang University
  • Sangdoo Yun NAVER Corporation
  • Jin Young Choi Seoul National University



Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.




How to Cite

Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019). Knowledge Distillation with Adversarial Samples Supporting Decision Boundary. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3771-3778.



AAAI Technical Track: Machine Learning