C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer

Authors

  • Dongxu Wei Department of Information Science and Electronic Engineering, Zhejiang University
  • Xiaowei Xu Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences
  • Haibin Shen Department of Information Science and Electronic Engineering, Zhejiang University
  • Kejie Huang Department of Information Science and Electronic Engineering, Zhejiang University

Keywords:

Computational Photography, Image & Video Synthesis

Abstract

Human video motion transfer (HVMT) aims to synthesize videos that one person imitates other persons' actions. Although existing GAN-based HVMT methods have achieved great success, they either fail to preserve appearance details due to the loss of spatial consistency between synthesized and exemplary images, or generate incoherent video results due to the lack of temporal consistency among video frames. In this paper, we propose Coarse-to-Fine Flow Warping Network (C2F-FWN) for spatial-temporal consistent HVMT. Particularly, C2F-FWN utilizes coarse-to-fine flow warping and Layout-Constrained Deformable Convolution (LC-DConv) to improve spatial consistency, and employs Flow Temporal Consistency (FTC) Loss to enhance temporal consistency. In addition, provided with multi-source appearance inputs, C2F-FWN can support appearance attribute editing with great flexibility and efficiency. Besides public datasets, we also collected a large-scale HVMT dataset named SoloDance for evaluation. Extensive experiments conducted on our SoloDance dataset and the iPER dataset show that our approach outperforms state-of-art HVMT methods in terms of both spatial and temporal consistency. Source code and the SoloDance dataset are available at https://github.com/wswdx/C2F-FWN.

Downloads

Published

2021-05-18

How to Cite

Wei, D., Xu, X., Shen, H., & Huang, K. (2021). C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4), 2852-2860. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16391

Issue

Section

AAAI Technical Track on Computer Vision III