SOIT: Segmenting Objects with Instance-Aware Transformers


  • Xiaodong Yu Hikvision Research Institute
  • Dahu Shi Hikvision Research Institute
  • Xing Wei Xi'an Jiaotong University
  • Ye Ren Hikvision Research Institute
  • Tingqun Ye Hikvision Research Institute
  • Wenming Tan Hikvision Research Institute



Computer Vision (CV)


This paper presents an end-to-end instance segmentation framework, termed SOIT, that Segments Objects with Instance-aware Transformers. Inspired by DETR, our method views instance segmentation as a direct set prediction problem and effectively removes the need for many hand-crafted components like RoI cropping, one-to-many label assignment, and non-maximum suppression (NMS). In SOIT, multiple queries are learned to directly reason a set of object embeddings of semantic category, bounding-box location, and pixel-wise mask in parallel under the global image context. The class and bounding-box can be easily embedded by a fixed-length vector. The pixel-wise mask, especially, is embedded by a group of parameters to construct a lightweight instance-aware transformer. Afterward, a full-resolution mask is produced by the instance-aware transformer without involving any RoI-based operation. Overall, SOIT introduces a simple single-stage instance segmentation framework that is both RoI- and NMS-free. Experimental results on the MS COCO dataset demonstrate that SOIT outperforms state-of-the-art instance segmentation approaches significantly. Moreover, the joint learning of multiple tasks in a unified query embedding can also substantially improve the detection performance. Code is available at




How to Cite

Yu, X., Shi, D., Wei, X., Ren, Y., Ye, T., & Tan, W. (2022). SOIT: Segmenting Objects with Instance-Aware Transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 3188-3196.



AAAI Technical Track on Computer Vision III