Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction

Authors

  • Zhenyu Zhang University of Chinese Academy of Sciences
  • Xiaobo Shu University of Chinese Academy of Sciences
  • Bowen Yu University of Chinese Academy of Sciences
  • Tingwen Liu University of Chinese Academy of Sciences
  • Jiapeng Zhao University of Chinese Academy of Sciences
  • Quangang Li University of Chinese Academy of Sciences
  • Li Guo University of Chinese Academy of Sciences

DOI:

https://doi.org/10.1609/aaai.v34i05.6509

Abstract

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e.g., type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

Downloads

Published

2020-04-03

How to Cite

Zhang, Z., Shu, X., Yu, B., Liu, T., Zhao, J., Li, Q., & Guo, L. (2020). Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 9620-9627. https://doi.org/10.1609/aaai.v34i05.6509

Issue

Section

AAAI Technical Track: Natural Language Processing