Conceptualized and Contextualized Gaussian Embedding

Authors

  • Chen Qian Tsinghua University
  • Fuli Feng National University of Singapore
  • Lijie Wen Tsinghua University
  • Tat-Seng Chua National university of Singapore

Keywords:

Lexical & Frame Semantics, Semantic Parsing

Abstract

Word embedding can represent a word as a point vector or a Gaussian distribution in high-dimensional spaces. Gaussian distribution is innately more expressive than point vector owing to the ability to additionally capture semantic uncertainties of words, and thus can express asymmetric relations among words more naturally (e.g., animal entails cat but not the reverse. However, previous Gaussian embedders neglect inner-word conceptual knowledge and lack tailored Gaussian contextualizer, leading to inferior performance on both intrinsic (context-agnostic) and extrinsic (context-sensitive) tasks. In this paper, we first propose a novel Gaussian embedder which explicitly accounts for inner-word conceptual units (sememes) to represent word semantics more precisely; during learning, we propose Gaussian Distribution Attention over Gaussian representations to adaptively aggregate multiple sememe distributions into a word distribution, which guarantees the Gaussian linear combination property. Additionally, we propose a Gaussian contextualizer to utilize outer-word contexts in a sentence, producing contextualized Gaussian representations for context-sensitive tasks. Extensive experiments on intrinsic and extrinsic tasks demonstrate the effectiveness of the proposed approach, achieving state-of-the-art performance with near 5.00% relative improvement.

Downloads

Published

2021-05-18

How to Cite

Qian, C., Feng, F., Wen, L., & Chua, T.-S. (2021). Conceptualized and Contextualized Gaussian Embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15), 13683-13691. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17613

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing II