Arbitrary Video Style Transfer via Multi-Channel Correlation

Authors

  • Yingying Deng School of Artificial Intelligence, University of Chinese Academy of Sciences, NLPR, Institute of Automation, Chinese Academy of Sciences, CASIA-LLvision Joint Lab
  • Fan Tang School of Artificial Intelligence, Jilin University
  • Weiming Dong School of Artificial Intelligence, University of Chinese Academy of Sciences, NLPR, Institute of Automation, Chinese Academy of Sciences, CASIA-LLvision Joint Lab
  • Haibin Huang Kuaishou Technology
  • Chongyang Ma Kuaishou Technology
  • Changsheng Xu School of Artificial Intelligence, University of Chinese Academy of Sciences, NLPR, Institute of Automation, Chinese Academy of Sciences, CASIA-LLvision Joint Lab

Keywords:

Computational Photography, Image & Video Synthesis

Abstract

Video style transfer is attracting increasing attention from the artificial intelligence community because of its numerous applications, such as augmented reality and animation production. Relative to traditional image style transfer, video style transfer presents new challenges, including how to effectively generate satisfactory stylized results for any specified style while maintaining temporal coherence across frames. Towards this end, we propose a Multi-Channel Correlation network (MCCNet), which can be trained to fuse exemplar style features and input content features for efficient style transfer while naturally maintaining the coherence of input videos to output videos. Specifically, MCCNet works directly on the feature space of style and content domain where it learns to rearrange and fuse style features on the basis of their similarity to content features. The outputs generated by MCC are features containing the desired style patterns that can further be decoded into images with vivid style textures. Moreover, MCCNet is also designed to explicitly align the features to input and thereby ensure that the outputs maintain the content structures and the temporal continuity. To further improve the performance of MCCNet under complex light conditions, we also introduce illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks. Code is available at https://github.com/diyiiyiii/MCCNet.

Downloads

Published

2021-05-18

How to Cite

Deng, Y., Tang, F., Dong, W., Huang, H., Ma, C., & Xu, C. (2021). Arbitrary Video Style Transfer via Multi-Channel Correlation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1210-1217. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16208

Issue

Section

AAAI Technical Track on Computer Vision I