A Model-Agnostic Heuristics for Selective Classification


  • Andrea Pugnana Scuola Normale Superiore, Pisa, IT
  • Salvatore Ruggieri University of Pisa, Pisa, IT




ML: Classification and Regression, ML: Ensemble Methods, PEAI: Safety, Robustness & Trustworthiness, RU: Other Foundations of Reasoning Under Uncertainty


Selective classification (also known as classification with reject option) conservatively extends a classifier with a selection function to determine whether or not a prediction should be accepted (i.e., trusted, used, deployed). This is a highly relevant issue in socially sensitive tasks, such as credit scoring. State-of-the-art approaches rely on Deep Neural Networks (DNNs) that train at the same time both the classifier and the selection function. These approaches are model-specific and computationally expensive. We propose a model-agnostic approach, as it can work with any base probabilistic binary classification algorithm, and it can be scalable to large tabular datasets if the base classifier is so. The proposed algorithm, called SCROSS, exploits a cross-fitting strategy and theoretical results for quantile estimation to build the selection function. Experiments on real-world data show that SCROSS improves over existing methods.




How to Cite

Pugnana, A., & Ruggieri, S. (2023). A Model-Agnostic Heuristics for Selective Classification. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8), 9461-9469. https://doi.org/10.1609/aaai.v37i8.26133



AAAI Technical Track on Machine Learning III