Attentive Tensor Product Learning


  • Qiuyuan Huang Microsoft Research
  • Li Deng Citadel
  • Dapeng Wu University of Florida
  • Chang Liu Citidel Securities
  • Xiaodong He JD AI Research



This paper proposes a novel neural architecture — Attentive Tensor Product Learning (ATPL) — to represent grammatical structures of natural language in deep learning models. ATPL exploits Tensor Product Representations (TPR), a structured neural-symbolic model developed in cognitive science, to integrate deep learning with explicit natural language structures and rules. The key ideas of ATPL are: 1) unsupervised learning of role-unbinding vectors of words via the TPR-based deep neural network; 2) the use of attention modules to compute TPR; and 3) the integration of TPR with typical deep learning architectures including long short-term memory and feedforward neural networks. The novelty of our approach lies in its ability to extract the grammatical structure of a sentence by using role-unbinding vectors, which are obtained in an unsupervised manner. Our ATPL approach is applied to 1) image captioning, 2) part of speech (POS) tagging, and 3) constituency parsing of a natural language sentence. The experimental results demonstrate the effectiveness of the proposed approach in all these three natural language processing tasks.




How to Cite

Huang, Q., Deng, L., Wu, D., Liu, C., & He, X. (2019). Attentive Tensor Product Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 1344-1351.



AAAI Technical Track: Cognitive Systems