Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems

Abstract

Recommendation systems have traditionally relied on short-term engagement signals, such as clicks and likes, to personalize content. However, these signals are often noisy, sparse, and insufficient for capturing whether a recommendation supports future return to the platform. We introduce Retentive Relevance, a novel content-level survey-based feedback measure that directly assesses users’ intent to return to the platform for similar content. Unlike other survey measures that focus on immediate satisfaction, Retentive Relevance targets forward-looking behavioral intentions and provides a stronger predictor of next-day retention. We validate Retentive Relevance using psychometric analyses suited to our single-item measures, establishing convergent, discriminant, and behavioral validity. Through large-scale offline modeling, we show that Retentive Relevance significantly outperforms both engagement signals and other survey measures in predicting next-day retention, especially for users with limited historical engagement. We develop a production-ready proxy model that integrates Retentive Relevance into the final stage of a multi-stage ranking system on a social media platform. Calibrated score adjustments based on this model yield improvements in engagement, retention, and content quality during a 14-day A/B experiment. This work links content-level user perceptions to short-horizon retention outcomes in production systems. We offer a scalable, user-centered approach with implications for responsible AI development.