Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

Authors

  • Zhizhong Han University of Maryland, College Park
  • Mingyang Shang Tsinghua University
  • Xiyang Wang Tsinghua University
  • Yu-Shen Liu Tsinghua University
  • Matthias Zwicker University of Maryland

DOI:

https://doi.org/10.1609/aaai.v33i01.3301126

Abstract

Jointly learning representations of 3D shapes and text is crucial to support tasks such as cross-modal retrieval or shape captioning. A recent method employs 3D voxels to represent 3D shapes, but this limits the approach to low resolutions due to the computational cost caused by the cubic complexity of 3D voxels. Hence the method suffers from a lack of detailed geometry. To resolve this issue, we propose Y2Seq2Seq, a view-based model, to learn cross-modal representations by joint reconstruction and prediction of view and word sequences. Specifically, the network architecture of Y2Seq2Seq bridges the semantic meaning embedded in the two modalities by two coupled “Y” like sequence-tosequence (Seq2Seq) structures. In addition, our novel hierarchical constraints further increase the discriminability of the cross-modal representations by employing more detailed discriminative information. Experimental results on cross-modal retrieval and 3D shape captioning show that Y2Seq2Seq outperforms the state-of-the-art methods.

Downloads

Published

2019-07-17

How to Cite

Han, Z., Shang, M., Wang, X., Liu, Y.-S., & Zwicker, M. (2019). Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 126-133. https://doi.org/10.1609/aaai.v33i01.3301126

Issue

Section

AAAI Technical Track: AI and the Web