GNN and LLM Insights: Multimodal Cues and Gender Disparities in Video Conversations

Abstract

As video content on online platforms continues to increase, understanding the complex aspects of interpersonal communication becomes crucial. Central to this exploration is the pressing issue of gender bias, which manifests in multimodal interactions through visual, vocal, or verbal cues. These interactions present challenges in extracting and interpreting the subtle cues that may point to underlying biases. To tackle these challenges, we introduce a semi-automatic extraction of features and knowledge from user-generated content on video web platforms. Using 1,091 unstructured multi-participant video conversations from Shark Tank, we examine whether the multimodal cues (e.g., emotions) of a conversational participant (e.g., entrepreneur) affect another participant (e.g., investor) differently due to gender biases. Our methodology employs advanced deep learning algorithms for cues extraction and leverages Graph Neural Networks to model the multi-participant conversations. To complement our findings, we utilize textual features extracted through our methodology and employ GPT-4 to simulate decision-making scenarios, thereby assessing its analytical capabilities and potential gender biases.