A Knowledge Graph Framework for Interpretable Video-Based Activity Recognition

Abstract

We propose two approaches for human activity recognition in videos that leverage knowledge graph representations. The first method constructs a Positional Encoding Knowledge Graph (PE-KG) by extracting objects and their spatial relationships from video keyframes, which are then analyzed using association rule mining. The second approach, termed Video KG, augments this representation by incorporating semantic cues from image captioning and affective insights from emotion detection with demographic analysis. The approach employs knowledge graph embeddings to capture spatiotemporal and contextual dependencies, leading to improved classification accuracy and enhanced interpretability on benchmarks such as the Kinetics dataset.