Interactive Speech and Noise Modeling for Speech Enhancement


  • Chengyu Zheng Communication University of China
  • Xiulian Peng Microsoft Research Asia
  • Yuan Zhang Communication University of China
  • Sriram Srinivasan Microsoft Corporation
  • Yan Lu Microsoft Research Asia



Speech & Signal Processing


Speech enhancement is challenging because of the diversity of background noise types. Most of the existing methods are focused on modelling the speech rather than the noise. In this paper, we propose a novel idea to model speech and noise simultaneously in a two-branch convolutional neural network, namely SN-Net. In SN-Net, the two branches predict speech and noise, respectively. Instead of information fusion only at the final output layer, interaction modules are introduced at several intermediate feature domains between the two branches to benefit each other. Such an interaction can leverage features learned from one branch to counteract the undesired part and restore the missing component of the other and thus enhance their discrimination capabilities. We also design a feature extraction module, namely residual-convolution-and-attention (RA), to capture the correlations along temporal and frequency dimensions for both the speech and the noises. Evaluations on public datasets show that the interaction module plays a key role in simultaneous modeling and the SN-Net outperforms the state-of-the-art by a large margin on various evaluation metrics. The proposed SN-Net also shows superior performance for speaker separation.




How to Cite

Zheng, C., Peng, X., Zhang, Y., Srinivasan, S., & Lu, Y. (2021). Interactive Speech and Noise Modeling for Speech Enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16), 14549-14557.



AAAI Technical Track on Speech and Natural Language Processing III