Evaluation of Concept Induction in Explainable AI Using Multiple Datasets

Abstract

Explainable AI aims to make artificial intelligence systems more transparent and trustworthy. A major challenge within this field involves deciphering the complex computational processes and understanding the activation patterns of hidden neurons. It was previously shown, in a single example, that an approach using formal logical reasoning in the form of so-called concept induction can meaningfully interpret hidden neuron activations in a Convolutional Neural Network (CNN) by assigning human-understandable labels. However, as this was only demonstrated for a single trained system, it is necessary to validate the approach using modified settings. In this paper, we replicate the results of the approach by employing an efficient automated pipeline which improved the processing speed and accuracy. We also extend the evaluation by selecting different training targets for the classifier. Our replication results are comparable to the previous ones and show that the approach can assign meaningful labels to individual neurons in the dense layer of a CNN, based on a statistical validation.