FedFAn: Federated Learning for Facial Animation

Abstract

We present FedFAn: a system for training audio-driven facial animation models across organisational silos via federated learning. AI model providers often serve companies who hold sensitive biometric and proprietary data which is used for model inference and would make valuable training data, but cannot be shared with the provider. In this paper we begin by outlining the problem of privacy in facial animation, defining three key groups for whom privacy is a consideration: trainers, managers and drivers. Trainers of avatar facial animation serve managers (such as studios) that hold valuable driver performance data and character avatars, but cannot reasonably centralise those assets with the trainer. FedFAn allows collaborative training by separating the identity-agnostic model components from the components that leak identifying information: the audio encoder is trained only during public pretraining and frozen during private federated learning, speaker style embeddings remain local to each client, and secure aggregation prevents the server from observing individual updates. We show that a model pretrained on the VOCASET dataset and then privately finetuned with federated learning on the HDTF dataset improves adaptation to unseen speakers, showing that decentralised data can improve cross-domain robustness, which benefits both trainers and managers. We evaluate the privacy limits of this architecture using membership inference attack, model-internal speaker identification, gradient leakage analysis, and a differential privacy feasibility study. These experiments show where speaker information (which can identify drivers) is carried, informing our model architecture and data handling approach.