Clustering Internet Memes with Metric Learning and Dynamic Modality Weighting

Abstract

This paper presents a two-stage metric learning approach for large-scale clustering of internet memes into pre-defined knowledge categories. We also introduce a novel dynamic modality weighting step to adaptively balance the influence of image and text attributes, which is previously unexplored in meme similarity tasks, and outperforms multimodal models such as CLIP. We train and evaluate the pipeline across 678,734 memes from KnowYourMeme.com and achieve an F1 score of 91% when assigning unseen memes from a different source to KnowYourMeme.com categories. Our proposed approach incorporates more meme types than prior research, enabling the alignment of individual memes to crucial knowledge sources for information retrieval tasks, with further applications in meme analysis, misinformation detection, hateful meme detection and internet cultural studies.