A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities
DOI:
https://doi.org/10.1609/aaai.v38i9.28871Keywords:
HAI: Applications, ML: Classification and RegressionAbstract
Multimodal Sentiment Analysis (MSA) has attracted widespread research attention recently. Most MSA studies are based on the assumption of modality completeness. However, many inevitable factors in real-world scenarios lead to uncertain missing modalities, which invalidate the fixed multimodal fusion approaches. To this end, we propose a Unified multimodal Missing modality self-Distillation Framework (UMDF) to handle the problem of uncertain missing modalities in MSA. Specifically, a unified self-distillation mechanism in UMDF drives a single network to automatically learn robust inherent representations from the consistent distribution of multimodal data. Moreover, we present a multi-grained crossmodal interaction module to deeply mine the complementary semantics among modalities through coarse- and fine-grained crossmodal attention. Eventually, a dynamic feature integration module is introduced to enhance the beneficial semantics in incomplete modalities while filtering the redundant information therein to obtain a refined and robust multimodal representation. Comprehensive experiments on three datasets demonstrate that our framework significantly improves MSA performance under both uncertain missing-modality and complete-modality testing conditions.Downloads
Published
2024-03-24
How to Cite
Li, M., Yang, D., Lei, Y., Wang, S., Wang, S., Su, L., Yang, K., Wang, Y., Sun, M., & Zhang, L. (2024). A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities. Proceedings of the AAAI Conference on Artificial Intelligence, 38(9), 10074-10082. https://doi.org/10.1609/aaai.v38i9.28871
Issue
Section
AAAI Technical Track on Humans and AI