Spectral Feature Augmentation for Graph Contrastive Learning and Beyond


  • Yifei Zhang The Chinese University of Hong Kong
  • Hao Zhu Australian National University Data61/CSIRO
  • Zixing Song The Chinese University of Hong Kong
  • Piotr Koniusz Data61/CSIRO Australian National University
  • Irwin King The Chinese University of Hong Kong




ML: Graph-based Machine Learning, DMKM: Graph Mining, Social Network Analysis & Community Mining, ML: Representation Learning, ML: Semi-Supervised Learning, ML: Unsupervised & Self-Supervised Learning


Although augmentations (e.g., perturbation of graph edges, image crops) boost the efficiency of Contrastive Learning (CL), feature level augmentation is another plausible, complementary yet not well researched strategy. Thus, we present a novel spectral feature argumentation for contrastive learning on graphs (and images). To this end, for each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power iteration regime which enjoys two valuable byproducts (under mere one or two iterations): (i) it partially balances spectrum of the feature map, and (ii) it injects the noise into rebalanced singular values of the feature map (spectral augmentation). For two views, we align these rebalanced feature maps as such an improved alignment step can focus more on less dominant singular values of matrices of both views, whereas the spectral augmentation does not affect the spectral angle alignment (singular vectors are not perturbed). We derive the analytical form for: (i) the incomplete power iteration to capture its spectrum-balancing effect, and (ii) the variance of singular values augmented implicitly by the noise. We also show that the spectral augmentation improves the generalization bound. Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines, and is complementary with other augmentation strategies and compatible with various contrastive losses.




How to Cite

Zhang, Y., Zhu, H., Song, Z., Koniusz, P., & King, I. (2023). Spectral Feature Augmentation for Graph Contrastive Learning and Beyond. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11289-11297. https://doi.org/10.1609/aaai.v37i9.26336



AAAI Technical Track on Machine Learning IV