Category-Aware Fine-Tuning and Cross-Age Transferability in Image Memorability Prediction

Abstract

Image memorability is highly consistent across observers, yet current vision models achieve only moderate accuracy and remain below human consistency. We study two questions: (i) whether making semantic category structure explicit during training improves prediction, and (ii) whether adult-trained predictors transfer to adolescents, and whether any gains from category-specific adaptation generalize across observers of different age. We compare a mixed-category model (All) with per-category fine-tuning (CatFT) for two pretrained backbones, MemNet (AlexNet-based CNN) and ViT-B/16 (Vision Transformer), each fine-tuned on MemCat under All and CatFT. Adult-trained models are evaluated on Memoir (adolescent labels) without additional training to assess transfer, and Grad-CAM is used to examine which regions drive predictions on the best model. On adults, category-aware training increases Spearman’s rho for both backbones (ViT-B/16: 0.548→0.592; MemNet: 0.429→0.477). Memorability prediction itself transfers across age even without category-specific fine-tuning (ViT-B/16: rho=0.456 with All), with a small additional adolescent gain from CatFT (to rho=0.471); MemNet remains stable on adolescents (rho=0.405 with or without CatFT). Grad-CAM highlights semantically meaningful regions for highly memorable images and more diffuse patterns for low-memorability images. Overall, incorporating category structure improves adult accuracy, cross-age generalization of memorability prediction is robust, and among the tested backbones, ViT-B/16 performs best, with CatFT providing modest transfer gains.