GPTR: Gestalt-Perception Transformer for Diagram Object Detection


  • Xin Hu Xi’an Jiaotong University
  • Lingling Zhang Xi'an Jiaotong University
  • Jun Liu Xi'an Jiaotong Univerisity
  • Jinfu Fan Tongji Univerisity
  • Yang You National University of Singapore
  • Yaqiang Wu Xi'an Jiaotong University Lenovo Research



CV: Applications, CMS: Simulating Humans, CV: Object Detection & Categorization, CV: Representation Learning for Vision


Diagram object detection is the key basis of practical applications such as textbook question answering. Because the diagram mainly consists of simple lines and color blocks, its visual features are sparser than those of natural images. In addition, diagrams usually express diverse knowledge, in which there are many low-frequency object categories in diagrams. These lead to the fact that traditional data-driven detection model is not suitable for diagrams. In this work, we propose a gestalt-perception transformer model for diagram object detection, which is based on an encoder-decoder architecture. Gestalt perception contains a series of laws to explain human perception, that the human visual system tends to perceive patches in an image that are similar, close or connected without abrupt directional changes as a perceptual whole object. Inspired by these thoughts, we build a gestalt-perception graph in transformer encoder, which is composed of diagram patches as nodes and the relationships between patches as edges. This graph aims to group these patches into objects via laws of similarity, proximity, and smoothness implied in these edges, so that the meaningful objects can be effectively detected. The experimental results demonstrate that the proposed GPTR achieves the best results in the diagram object detection task. Our model also obtains comparable results over the competitors in natural image object detection.




How to Cite

Hu, X., Zhang, L., Liu, J., Fan, J., You, Y., & Wu, Y. (2023). GPTR: Gestalt-Perception Transformer for Diagram Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1), 899-907.



AAAI Technical Track on Computer Vision I