Using k-Way Co-Occurrences for Learning Word Embeddings

Authors

  • Danushka Bollegala The University of Liverpool
  • Yuichi Yoshida National Institute of Informatics
  • Ken-ichi Kawarabayashi National Institute of Informatics

Keywords:

Word Embeddings, k-way co-occurrences

Abstract

Co-occurrences between two words provide useful insights into the semantics of those words.Consequently, numerous prior work on word embedding learning has used co-occurrences between two wordsas the training signal for learning word embeddings.However, in natural language texts it is common for multiple words to be related and co-occurring in the same context.We extend the notion of co-occurrences to cover k(≥2)-way co-occurrences among a set of k-words.Specifically, we prove a theoretical relationship between the joint probability of k(≥2) words, and the sum of l_2 norms of their embeddings. Next, we propose a learning objective motivated by our theoretical resultthat utilises k-way co-occurrences for learning word embeddings.Our experimental results show that the derived theoretical relationship does indeed hold empirically, anddespite data sparsity, for some smaller k(≤5) values, k-way embeddings perform comparably or better than 2-way embeddings in a range of tasks.

Downloads

Published

2018-04-27

How to Cite

Bollegala, D., Yoshida, Y., & Kawarabayashi, K.- ichi. (2018). Using k-Way Co-Occurrences for Learning Word Embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/12010