TY - JOUR AU - Martínez-Plumed, Fernando AU - Castellano, David AU - Monserrat-Aranda, Carlos AU - Hernández-Orallo, José PY - 2022/06/28 Y2 - 2024/03/28 TI - When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 36 IS - 7 SE - AAAI Technical Track on Machine Learning II DO - 10.1609/aaai.v36i7.20739 UR - https://ojs.aaai.org/index.php/AAAI/article/view/20739 SP - 7719-7727 AB - One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications. ER -