Learning Prediction Intervals for Model Performance

Authors

  • Benjamin Elder IBM Research
  • Matthew Arnold IBM Research
  • Anupama Murthi IBM Research
  • Jiří Navrátil IBM Research

Keywords:

Calibration & Uncertainty Quantification

Abstract

Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use.

Downloads

Published

2021-05-18

How to Cite

Elder, B., Arnold, M., Murthi, A., & Navrátil, J. (2021). Learning Prediction Intervals for Model Performance. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7305-7313. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16897

Issue

Section

AAAI Technical Track on Machine Learning I