JAKET: Joint Pre-training of Knowledge Graph and Language Understanding


  • Donghan Yu Carnegie Mellon University
  • Chenguang Zhu Microsoft
  • Yiming Yang Carnegie Mellon University
  • Michael Zeng Microsoft




Speech & Natural Language Processing (SNLP)


Knowledge graphs (KGs) contain rich information about world knowledge, entities, and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experiment results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.




How to Cite

Yu, D., Zhu, C., Yang, Y., & Zeng, M. (2022). JAKET: Joint Pre-training of Knowledge Graph and Language Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11630-11638. https://doi.org/10.1609/aaai.v36i10.21417



AAAI Technical Track on Speech and Natural Language Processing