Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders

Authors

  • Abhishek Banerjee University of Maryland, College Park
  • Uttaran Bhattacharya University of Maryland, College Park
  • Aniket Bera University of Maryland, College Park

DOI:

https://doi.org/10.1609/aaai.v36i1.19873

Keywords:

Cognitive Modeling & Cognitive Systems (CMS)

Abstract

We present a novel generalized zero-shot algorithm to recognize perceived emotions from gestures. Our task is to map gestures to novel emotion categories not encountered in training. We introduce an adversarial autoencoder-based representation learning that correlates 3D motion-captured gesture sequences with the vectorized representation of the natural-language perceived emotion terms using word2vec embeddings. The language-semantic embedding provides a representation of the emotion label space, and we leverage this underlying distribution to map the gesture sequences to the appropriate categorical emotion labels. We train our method using a combination of gestures annotated with known emotion terms and gestures not annotated with any emotions. We evaluate our method on the MPI Emotional Body Expressions Database (EBEDB) and obtain an accuracy of 58.43%. We see an improvement in performance compared to current state-of-the-art algorithms for generalized zero-shot learning by an absolute 25-27%. We also demonstrate our approach on publicly available online videos and movie scenes, where the actors' pose has been extracted and map to their respective emotive states.

Downloads

Published

2022-06-28

How to Cite

Banerjee, A., Bhattacharya, U., & Bera, A. (2022). Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1), 3-10. https://doi.org/10.1609/aaai.v36i1.19873

Issue

Section

AAAI Technical Track on Cognitive Modeling & Cognitive Systems