Obtaining Well Calibrated Probabilities Using Bayesian Binning

Authors

  • Mahdi Pakdaman Naeini University of Pittsburgh
  • Gregory Cooper University of Pittsburgh
  • Milos Hauskrecht University of Pittsburgh

DOI:

https://doi.org/10.1609/aaai.v29i1.9602

Keywords:

Bayesian binning, classifier calibration, accurate probability, calibrated probability, Bayesian Scoring

Abstract

Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in artificial intelligence. In this paper we present a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) which addresses key limitations of existing calibration methods. The method post processes the output of a binary classification algorithm; thus, it can be readily combined with many existing classification algorithms. The method is computationally tractable, and empirically accurate, as evidenced by the set of experiments reported here on both real and simulated datasets.

Downloads

Published

2015-02-21

How to Cite

Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015). Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). https://doi.org/10.1609/aaai.v29i1.9602

Issue

Section

Main Track: Novel Machine Learning Algorithms