Learning to Pre-train Graph Neural Networks

Authors

  • Yuanfu Lu Beijing University of Posts and Telecommunications WeChat Search Application Department, Tencent Inc. China
  • Xunqiang Jiang Beijing University of Posts and Telecommunications
  • Yuan Fang Singapore Management University
  • Chuan Shi Beijing University of Posts and Telecommunications Peng Cheng Laboratory, Shenzhen, China

DOI:

https://doi.org/10.1609/aaai.v35i5.16552

Keywords:

Graph Mining, Social Network Analysis & Community, Linked Open Data, Knowledge Graphs & KB Completio, (Deep) Neural Network Algorithms

Abstract

Graph neural networks (GNNs) have become the defacto standard for representation learning on graphs, which derive effective node representations by recursively aggregating information from graph neighborhoods. While GNNs can be trained from scratch, pre-training GNNs to learn transferable knowledge for downstream tasks has recently been demonstrated to improve the state of the art. However, conventional GNN pre-training methods follow a two-step paradigm: 1) pre-training on abundant unlabeled data and 2) fine-tuning on downstream labeled data, between which there exists a significant gap due to the divergence of optimization objectives in the two steps. In this paper, we conduct an analysis to show the divergence between pre-training and fine-tuning, and to alleviate such divergence, we propose L2P-GNN, a self-supervised pre-training strategy for GNNs. The key insight is that L2P-GNN attempts to learn how to fine-tune during the pre-training process in the form of transferable prior knowledge. To encode both local and global information into the prior, L2P-GNN is further designed with a dual adaptation mechanism at both node and graph levels. Finally, we conduct a systematic empirical study on the pre-training of various GNN models, using both a public collection of protein graphs and a new compilation of bibliographic graphs for pre-training. Experimental results show that L2P-GNN is capable of learning effective and transferable prior knowledge that yields powerful representations for downstream tasks. (Code and datasets are available at https://github.com/rootlu/L2P-GNN.)

Downloads

Published

2021-05-18

How to Cite

Lu, Y., Jiang, X., Fang, Y., & Shi, C. (2021). Learning to Pre-train Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5), 4276-4284. https://doi.org/10.1609/aaai.v35i5.16552

Issue

Section

AAAI Technical Track on Data Mining and Knowledge Management