Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing


  • Dong Zhang Soochow University
  • Xincheng Ju Soochow University
  • Wei Zhang Alibaba Group
  • Junhui Li Soochow University
  • Shoushan Li Soochow University
  • Qiaoming Zhu Soochow University
  • Guodong Zhou Soochow University



Text Classification & Sentiment Analysis, Affective Computing


As an important research issue in affective computing community, multi-modal emotion recognition has become a hot topic in the last few years. However, almost all existing studies perform multiple binary classification for each emotion with focus on complete time series data. In this paper, we focus on multi-modal emotion recognition in a multi-label scenario. In this scenario, we consider not only the label-to-label dependency, but also the feature-to-label and modality-to-label dependencies. Particularly, we propose a heterogeneous hierarchical message passing network to effectively model above dependencies. Furthermore, we propose a new multi-modal multi-label emotion dataset based on partial time-series content to show predominant generalization of our model. Detailed evaluation demonstrates the effectiveness of our approach.




How to Cite

Zhang, D., Ju, X., Zhang, W., Li, J., Li, S., Zhu, Q., & Zhou, G. (2021). Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16), 14338-14346.



AAAI Technical Track on Speech and Natural Language Processing III