Investigating LLM-Generated Feature Importance for XAI

Abstract

Complex machine learning models, such as XGBoost, are often used on tabular data, but explaining their predictions is a non-trivial task that requires the setup and configuration of post-hoc explainers such as SHAP or LIME. These explainers need access to the model and technical expertise, which are often unavailable when a model is proprietary or only accessible through a restricted interface. Large Language Models (LLMs) could offer an alternative, as prior work has shown that LLMs can rank feature importance without access to the model’s internals, but only on small binary classification datasets, so it is unclear when their rankings agree with established explainers.

To address this gap, this thesis investigates under what conditions LLM-generated feature importance rankings agree with SHAP and LIME. To answer this, we trained XGBoost models on three datasets across different task types and dimensionalities: California Housing, Wine Quality, and Taiwan Bankruptcy Prediction. We then prompted two open-weight LLMs, Gemma 4 (31B) and GPT-OSS (120B), to rank which features are driving 100 individual predictions per dataset. For the prompting, we used five prompt templates derived from three approaches: text-driven prompts that provide feature names and task context, data-driven prompts that provide anonymized perturbed samples, and a novel mixed neighbors prompt that combines feature names with real contrastive neighboring samples. The rankings were compared against rankings from SHAP and LIME using Spearman correlation and top-k overlap.

We found that alignment is mainly dependent on the dataset characteristics and the information included in the prompt. The best setup on California Housing, mixed neighbors with Gemma, came close to the agreement between SHAP and LIME themselves (a Spearman correlation of 0.693 compared to 0.772). On Wine Quality, where SHAP and LIME are in slight agreement (0.226 Spearman correlation), most setups surpassed their agreement, with a text-driven prompt reaching 0.645 against SHAP. On Taiwan Bankruptcy Prediction, all setups stayed below the agreement between SHAP and LIME (at best 0.305 compared to 0.414), with data-driven prompts performing best. Across all three datasets, the choice of prompt template affected agreement more than the choice of LLM.

However, text-driven rankings often missed the most important feature of individual predictions, suggesting that their agreement comes largely from the LLM’s own domain knowledge rather than the sample’s feature values. Overall, LLM-generated rankings can offer a quick, rough estimate of which features are predictive, but are not able to completely replace traditional post-hoc explainers. In practice, LLMs are best suited as a first check when model access is restricted, while high-stakes decisions still require explanations computed from the model itself.