When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty
DOI:
https://doi.org/10.1609/aaai.v36i7.20739Keywords:
Machine Learning (ML)Abstract
One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.Downloads
Published
2022-06-28
How to Cite
Martínez-Plumed, F., Castellano, D., Monserrat-Aranda, C., & Hernández-Orallo, J. (2022). When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7719-7727. https://doi.org/10.1609/aaai.v36i7.20739
Issue
Section
AAAI Technical Track on Machine Learning II