Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

Authors

  • Dong Zhang Soochow University
  • Xincheng Ju Soochow University
  • Wei Zhang Alibaba Group
  • Junhui Li Soochow University
  • Shoushan Li Soochow University
  • Qiaoming Zhu Soochow University
  • Guodong Zhou Soochow University

DOI:

https://doi.org/10.1609/aaai.v35i16.17686

Keywords:

Text Classification & Sentiment Analysis, Affective Computing

Abstract

As an important research issue in affective computing community, multi-modal emotion recognition has become a hot topic in the last few years. However, almost all existing studies perform multiple binary classification for each emotion with focus on complete time series data. In this paper, we focus on multi-modal emotion recognition in a multi-label scenario. In this scenario, we consider not only the label-to-label dependency, but also the feature-to-label and modality-to-label dependencies. Particularly, we propose a heterogeneous hierarchical message passing network to effectively model above dependencies. Furthermore, we propose a new multi-modal multi-label emotion dataset based on partial time-series content to show predominant generalization of our model. Detailed evaluation demonstrates the effectiveness of our approach.

Downloads

Published

2021-05-18

How to Cite

Zhang, D., Ju, X., Zhang, W., Li, J., Li, S., Zhu, Q., & Zhou, G. (2021). Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16), 14338-14346. https://doi.org/10.1609/aaai.v35i16.17686

Issue

Section

AAAI Technical Track on Speech and Natural Language Processing III