On Estimating Recommendation Evaluation Metrics under Sampling


  • Ruoming Jin Kent State University
  • Dong Li Kent State University
  • Benjamin Mudrak Kent State University
  • Jing Gao iLambda
  • Zhi Liu iLambda




Recommender Systems & Collaborative Filtering


Since the recent studies (KDD'20) done by Krichene and Rendle on the sampling based top-k evaluation metric for recommendation, there have been a lot of debate on the validity of using sampling for evaluating recommendation algorithms. Though their work and the recent work done by Li et. al. (KDD'20) have proposed some basic approach for mapping the sampling based metrics to their counter-part in the global evaluation which uses the entire dataset, there is still lack of understanding how sampling should be used for recommendation evaluation, and the proposed approaches either are rather ad-hoc or can only work on simple metrics, like Recall/Hit-Ratio. In this paper, we introduce some principled approach to derive the estimators of top-k metric based on sampling. Our approaches utilize the weighted MLE and maximal entropy approach to recover the global rank distribution and then utilize that for estimation. The experimental results shows significant advantages of using our approaches for evaluating recommendation algorithms based on top-k metrics.




How to Cite

Jin, R., Li, D., Mudrak, B., Gao, J., & Liu, Z. (2021). On Estimating Recommendation Evaluation Metrics under Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5), 4147-4154. https://doi.org/10.1609/aaai.v35i5.16537



AAAI Technical Track on Data Mining and Knowledge Management