Orthogonal Graph Neural Networks

Authors

  • Kai Guo jilin university
  • Kaixiong Zhou Rice University
  • Xia Hu Rice University
  • Yu Li Jilin University
  • Yi Chang Jilin University
  • Xin Wang Jilin University

DOI:

https://doi.org/10.1609/aaai.v36i4.20316

Keywords:

Data Mining & Knowledge Management (DMKM)

Abstract

Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations. These models rely on message passing and feature transformation functions to encode the structural and feature information from neighbors. However, stacking more convolutional layers significantly decreases the performance of GNNs. Most recent studies attribute this limitation to the over-smoothing issue, where node embeddings converge to indistinguishable vectors. Through a number of experimental observations, we argue that the main factor degrading the performance is the unstable forward normalization and backward gradient resulted from the improper design of the feature transformation, especially for shallow GNNs where the over-smoothing has not happened. Therefore, we propose a novel orthogonal feature transformation, named Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance. Specifically, we maintain the orthogonality of the feature transformation comprehensively from three perspectives, namely hybrid weight initialization, orthogonal transformation, and orthogonal regularization. By equipping the existing GNNs (e.g. GCN, JKNet, GCNII) with Ortho-GConv, we demonstrate the generality of the orthogonal feature transformation to enable stable training, and show its effectiveness for node and graph classification tasks.

Downloads

Published

2022-06-28

How to Cite

Guo, K., Zhou, K., Hu, X., Li, Y., Chang, Y., & Wang, X. (2022). Orthogonal Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4), 3996-4004. https://doi.org/10.1609/aaai.v36i4.20316

Issue

Section

AAAI Technical Track on Data Mining and Knowledge Management