Proper Proxy Scoring Rules


  • Jens Witkowski ETH Zurich
  • Pavel Atanasov Pytho LLC
  • Lyle Ungar University of Pennsylvania
  • Andreas Krause ETH Zurich



Information Elicitation, Forecasting, Proper Scoring Rules, Peer Prediction, Mechanism Design, Incentive Schemes


Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.




How to Cite

Witkowski, J., Atanasov, P., Ungar, L., & Krause, A. (2017). Proper Proxy Scoring Rules. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).



AAAI Technical Track: Game Theory and Economic Paradigms