MicroAST: Towards Super-fast Ultra-Resolution Arbitrary Style Transfer

Authors

  • Zhizhong Wang Zhejiang University
  • Lei Zhao Zhejiang University
  • Zhiwen Zuo Zhejiang University
  • Ailin Li Zhejiang University
  • Haibo Chen Zhejiang University
  • Wei Xing Zhejiang University
  • Dongming Lu Zhejiang University

DOI:

https://doi.org/10.1609/aaai.v37i3.25374

Keywords:

CV: Computational Photography, Image & Video Synthesis, CV: Applications, APP: Art/Music/Creativity, ML: Deep Generative Models & Autoencoders

Abstract

Arbitrary style transfer (AST) transfers arbitrary artistic styles onto content images. Despite the recent rapid progress, existing AST methods are either incapable or too slow to run at ultra-resolutions (e.g., 4K) with limited resources, which heavily hinders their further applications. In this paper, we tackle this dilemma by learning a straightforward and lightweight model, dubbed MicroAST. The key insight is to completely abandon the use of cumbersome pre-trained Deep Convolutional Neural Networks (e.g., VGG) at inference. Instead, we design two micro encoders (content and style encoders) and one micro decoder for style transfer. The content encoder aims at extracting the main structure of the content image. The style encoder, coupled with a modulator, encodes the style image into learnable dual-modulation signals that modulate both intermediate features and convolutional filters of the decoder, thus injecting more sophisticated and flexible style signals to guide the stylizations. In addition, to boost the ability of the style encoder to extract more distinct and representative style signals, we also introduce a new style signal contrastive loss in our model. Compared to the state of the art, our MicroAST not only produces visually superior results but also is 5-73 times smaller and 6-18 times faster, for the first time enabling super-fast (about 0.5 seconds) AST at 4K ultra-resolutions.

Downloads

Published

2023-06-26

How to Cite

Wang, Z., Zhao, L., Zuo, Z., Li, A., Chen, H., Xing, W., & Lu, D. (2023). MicroAST: Towards Super-fast Ultra-Resolution Arbitrary Style Transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 37(3), 2742-2750. https://doi.org/10.1609/aaai.v37i3.25374

Issue

Section

AAAI Technical Track on Computer Vision III