Video as Conditional Graph Hierarchy for Multi-Granular Question Answering


  • Junbin Xiao National University of Singapore
  • Angela Yao National University of Singapore
  • Zhiyuan Liu National University of Singapore
  • Yicong Li National University of Singapore
  • Wei Ji National University of Singapore
  • Tat-Seng Chua National University of Singapore



Computer Vision (CV), Cognitive Modeling & Cognitive Systems (CMS), Knowledge Representation And Reasoning (KRR)


Video question answering requires the models to understand and reason about both the complex video and language data to correctly derive the answers. Existing efforts have been focused on designing sophisticated cross-modal interactions to fuse the information from two modalities, while encoding the video and question holistically as frame and word sequences. Despite their success, these methods are essentially revolving around the sequential nature of video- and question-contents, providing little insight to the problem of question-answering and lacking interpretability as well. In this work, we argue that while video is presented in frame sequence, the visual elements (e.g., objects, actions, activities and events) are not sequential but rather hierarchical in semantic space. To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues. Despite the simplicity, our extensive experiments demonstrate the superiority of such conditional hierarchical graph architecture, with clear performance improvements over prior methods and also better generalization across different type of questions. Further analyses also demonstrate the model's reliability as it shows meaningful visual-textual evidences for the predicted answers.




How to Cite

Xiao, J., Yao, A., Liu, Z., Li, Y., Ji, W., & Chua, T.-S. (2022). Video as Conditional Graph Hierarchy for Multi-Granular Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 2804-2812.



AAAI Technical Track on Computer Vision III