Text-Guided Graph Neural Networks for Referring 3D Instance Segmentation

Authors

  • Pin-Hao Huang Institute of Information Science, Academia Sinica, Taiwan
  • Han-Hung Lee Department of Computer Science, National Tsing Hua University, Taiwan Institute of Information Science, Academia Sinica, Taiwan
  • Hwann-Tzong Chen Department of Computer Science, National Tsing Hua University, Taiwan Aeolus Robotics
  • Tyng-Luh Liu Institute of Information Science, Academia Sinica, Taiwan Taiwan AI Labs

Keywords:

3D Computer Vision

Abstract

This paper addresses a new task called referring 3D instance segmentation, which aims to segment out the target instance in a 3D scene given a query sentence. Previous work on scene understanding has explored visual grounding with natural language guidance, yet the emphasis is mostly constrained on images and videos. We propose a Text-guided Graph Neural Network (TGNN) for referring 3D instance segmentation on point clouds. Given a query sentence and the point cloud of a 3D scene, our method learns to extract per-point features and predicts an offset to shift each point toward its object center. Based on the point features and the offsets, we cluster the points to produce fused features and coordinates for the candidate objects. The resulting clusters are modeled as nodes in a Graph Neural Network to learn the representations that encompass the relation structure for each candidate object. The GNN layers leverage each object's features and its relations with neighbors to generate an attention heatmap for the input sentence expression. Finally, the attention heatmap is used to "guide" the aggregation of information from neighborhood nodes. Our method achieves state-of-the-art performance on referring 3D instance segmentation and 3D localization on ScanRefer, Nr3D, and Sr3D benchmarks, respectively.

Downloads

Published

2021-05-18

How to Cite

Huang, P.-H., Lee, H.-H., Chen, H.-T., & Liu, T.-L. (2021). Text-Guided Graph Neural Networks for Referring 3D Instance Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1610-1618. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16253

Issue

Section

AAAI Technical Track on Computer Vision I