Object-Centric Image Generation from Layouts

Authors

  • Tristan Sylvain Mila University of Montreal
  • Pengchuan Zhang Microsoft Research AI
  • Yoshua Bengio Mila University of Montreal CIFAR Senior Fellow
  • R Devon Hjelm Microsoft Research Mila
  • Shikhar Sharma Microsoft Turing

Keywords:

Computational Photography, Image & Video Synthesis

Abstract

We begin with the hypothesis that a model must be able to understand individual objects and relationships between objects in order to generate complex scenes with multiple objects well. Our layout-to-image-generation method, which we call Object-Centric Generative Adversarial Network (or OC-GAN), relies on a novel Scene-Graph Similarity Module (SGSM). The SGSM learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We also propose changes to the conditioning mechanism of the generator that enhance its object instance-awareness. Apart from improving image quality, our contributions mitigate two failure modes in previous approaches: (1) spurious objects being generated without corresponding bounding boxes in the layout, and (2) overlapping bounding boxes in the layout leading to merged objects in images. Extensive quantitative evaluation and ablation studies demonstrate the impact of our contributions, with our model outperforming previous state-of-the-art approaches on both the COCO-Stuff and Visual Genome datasets. Finally, we address an important limitation of evaluation metrics used in previous works by introducing SceneFID -- an object-centric adaptation of the popular Fréchet Inception Distance metric, that is better suited for multi-object images.

Downloads

Published

2021-05-18

How to Cite

Sylvain, T., Zhang, P., Bengio, Y., Hjelm, R. D., & Sharma, S. (2021). Object-Centric Image Generation from Layouts. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 2647-2655. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16368

Issue

Section

AAAI Technical Track on Computer Vision II