Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives


  • Yanzhao Zhang Beihang University
  • Richong Zhang Beihang University
  • Samuel Mensah University of Sheffield
  • Xudong Liu Beihang University
  • Yongyi Mao University of Ottawa




Speech & Natural Language Processing (SNLP)


Unsupervised sentence representation learning is a fundamental problem in natural language processing. Recently, contrastive learning has made great success on this task. Existing constrastive learning based models usually apply random sampling to select negative examples for training. Previous work in computer vision has shown that hard negative examples help contrastive learning to achieve faster convergency and better optimization for representation learning. However, the importance of hard negatives in contrastive learning for sentence representation is yet to be explored. In this study, we prove that hard negatives are essential for maintaining strong gradient signals in the training process while random sampling negative examples is ineffective for sentence representation. Accordingly, we present a contrastive model, MixCSE, that extends the current state-of-the-art SimCSE by continually constructing hard negatives via mixing both positive and negative features. The superior performance of the proposed approach is demonstrated via empirical studies on Semantic Textual Similarity datasets and Transfer task datasets.




How to Cite

Zhang, Y., Zhang, R., Mensah, S., Liu, X., & Mao, Y. (2022). Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11730-11738. https://doi.org/10.1609/aaai.v36i10.21428



AAAI Technical Track on Speech and Natural Language Processing