BEAM

Abstract

The rationale behind deep neural network (DNN) predictions is often difficult to understand by humans. While EXplainable AI (XAI) methods aim at improving the explainability of DNNs, their explanations need to be reliably evaluated and compared to ground truth (GT) explanations. Existing evaluation protocols often suffer from unreliability due to the black-box nature of DNNs and resulting lack of GTs. Consequently, many existing works resorted to guessed GTs. In this paper, we shift from guessed GTs and propose a novel evaluation framework for benchmarking XAI attribution methods, which consists of a carefully designed synthetic convolutional or vision transformer image classification model accompanied by synthetic GTs. It enables precise representation of input node contributions. We also propose high-fidelity metrics to quantify the alignment between explanations of the investigated XAI method and GTs. We investigate this approach by constructing synthetic image classification models and benchmarking several widely used XAI attribution methods. Our results provide essential insights into the performance of XAI methods including 1) the imbalance in explanation fidelity between positively and negatively contributing pixels, 2) all instances of a concept being highlighted even when only a subset contributes to model output, 3) non-linear relationships between input and output being incorrectly explained, and 4) XAI methods not explaining that an input belongs to a class due to absence of a feature. Finally, our framework allows for exact evaluation of XAI methods that can be subsequently deployed for real-world task explanations. We release code and materials at https://github.com/rbrandt1/BEAM.