Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations

Authors

  • Fei Sun Institute of Computing Technology, Chinese Academy of Sciences
  • Jiafeng Guo Institute of Computing Technology, Chinese Academy of Sciences
  • Yanyan Lan Institute of Computing Technology, Chinese Academy of Sciences
  • Jun Xu Institute of Computing Technology, Chinese Academy of Sciences
  • Xueqi Cheng Institute of Computing Technology, Chinese Academy of Sciences

DOI:

https://doi.org/10.1609/aaai.v30i1.10338

Keywords:

Word Representation, Distributional Hypothesis, Morphology, Phrase Representation

Abstract

Distributional hypothesis lies in the root of most existing word representation models by inferring word meaning from its external contexts. However, distributional models cannot handle rare and morphologically complex words very well and fail to identify some fine-grained linguistic regularity as they are ignoring the word forms. On the contrary, morphology points out that words are built from some basic units, i.e., morphemes. Therefore, the meaning and function of such rare words can be inferred from the words sharing the same morphemes, and many syntactic relations can be directly identified based on the word forms. However, the limitation of morphology is that it cannot infer the relationship between two words that do not share any morphemes. Considering the advantages and limitations of both approaches, we propose two novel models to build better word representations by modeling both external contexts and internal morphemes in a jointly predictive way, called BEING and SEING. These two models can also be extended to learn phrase representations according to the distributed morphology theory. We evaluate the proposed models on similarity tasks and analogy tasks. The results demonstrate that the proposed models can outperform state-of-the-art models significantly on both word and phrase representation learning.

Downloads

Published

2016-03-05

How to Cite

Sun, F., Guo, J., Lan, Y., Xu, J., & Cheng, X. (2016). Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10338

Issue

Section

Technical Papers: NLP and Machine Learning