Reference-Based Speech Enhancement via Feature Alignment and Fusion Network


  • Huanjing Yue Tianjin University
  • Wenxin Duo Tianjin University
  • Xiulian Peng Individual
  • Jingyu Yang Tianjin University



Speech & Natural Language Processing (SNLP)


Speech enhancement aims at recovering a clean speech from a noisy input, which can be classified into single speech enhancement and personalized speech enhancement. Personalized speech enhancement usually utilizes the speaker identity extracted from the noisy speech itself (or a clean reference speech) as a global embedding to guide the enhancement process. Different from them, we observe that the speeches of the same speaker are correlated in terms of frame-level short-time Fourier Transform (STFT) spectrogram. Therefore, we propose reference-based speech enhancement via a feature alignment and fusion network (FAF-Net). Given a noisy speech and a clean reference speech spoken by the same speaker, we first propose a feature level alignment strategy to warp the clean reference with the noisy speech in frame level. Then, we fuse the reference feature with the noisy feature via a similarity-based fusion strategy. Finally, the fused features are skipped connected to the decoder, which generates the enhanced results. Experimental results demonstrate that the performance of the proposed FAF-Net is close to state-of-the-art speech enhancement methods on both DNS and Voice Bank+DEMAND datasets. Our code is available at




How to Cite

Yue, H., Duo, W., Peng, X., & Yang, J. (2022). Reference-Based Speech Enhancement via Feature Alignment and Fusion Network. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11648-11656.



AAAI Technical Track on Speech and Natural Language Processing