Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling


  • Jia-Qi Yang Nanjing University
  • Xiang Li Alibaba Group
  • Shuguang Han Alibaba Group
  • Tao Zhuang Alibaba Group
  • De-Chuan Zhan Nanjing University
  • Xiaoyi Zeng Alibaba Group
  • Bin Tong Alibaba Group



Recommender Systems & Collaborative Filtering, Business/Marketing/Advertising/E-commerce


Conversion rate (CVR) prediction is one of the most critical tasks for digital display advertising. Commercial systems often require to update models in an online learning manner to catch up with the evolving data distribution. However, conversions usually do not happen immediately after user clicks. This may result in inaccurate labeling, which is called delayed feedback problem. In previous studies, delayed feedback problem is handled either by waiting positive label for a long period of time, or by consuming the negative sample on its arrival and then insert a positive duplicate when conversion happens later. Indeed, there is a trade-off between waiting for more accurate labels and utilizing fresh data, which is not considered in existing works. To strike a balance in this trade-off, we propose Elapsed-Time Sampling Delayed Feedback Model (ES-DFM), which models the relationship between the observed conversion distribution and the true conversion distribution. Then we optimize the expectation of true conversion distribution via importance sampling under the elapsed-time sampling distribution. We further estimate the importance weight for each instance, which is used as the weight of loss function in CVR prediction. To demonstrate the effectiveness of ES-DFM, we conduct extensive experiments on a public data and a private industrial dataset. Experimental results confirm that our method consistently outperforms the previous state-of-the-art results.




How to Cite

Yang, J.-Q., Li, X., Han, S., Zhuang, T., Zhan, D.-C., Zeng, X., & Tong, B. (2021). Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5), 4582-4589.



AAAI Technical Track on Data Mining and Knowledge Management