Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Authors

  • Lincheng Li Netease Fuxi AI Lab
  • Suzhen Wang Netease Fuxi AI Lab
  • Zhimeng Zhang Netease Fuxi AI Lab
  • Yu Ding Netease Fuxi AI Lab
  • Yixing Zheng Netease Fuxi AI Lab
  • Xin Yu University of Technology Sydney
  • Changjie Fan Netease Fuxi AI Lab

Keywords:

Computational Photography, Image & Video Synthesis

Abstract

In this paper, we propose a novel text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions in accordance with contextual sentiments as well as speech rhythm and pauses. To be specific, our framework consists of a speaker-independent stage and a speaker-specific stage. In the speaker-independent stage, we design three parallel networks to generate animation parameters of the mouth, upper face, and head from texts, separately. In the speaker-specific stage, we present a 3D face model guided attention network to synthesize videos tailored for different individuals. It takes the animation parameters as input and exploits an attention mask to manipulate facial expression changes for the input individuals. Furthermore, to better establish authentic correspondences between visual motions (i.e., facial expression changes and head movements) and audios, we leverage a high-accuracy motion capture dataset instead of relying on long videos of specific individuals. After attaining the visual and audio correspondences, we can effectively train our network in an end-to-end fashion. Extensive experiments on qualitative and quantitative results demonstrate that our algorithm achieves high-quality photo-realistic talking-head videos including various facial expressions and head motions according to speech rhythms and outperforms the state-of-the-art.

Downloads

Published

2021-05-18

How to Cite

Li, L., Wang, S., Zhang, Z., Ding, Y., Zheng, Y., Yu, X., & Fan, C. (2021). Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 1911-1920. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16286

Issue

Section

AAAI Technical Track on Computer Vision II