When to trust an explanation?
by
Abstract:
Explainable AI (XAI) aims at the extension or substitution of black-box machine learning models by components, which are comprehensible for humans. Prominent examples include prototype-based approaches such as counterfactual explanations or feature importance measures such as SHAP. Besides the increasing number of XAI methods, however, it is unclear, what information is included in these components, why different methods lead to different results, and in which sense humans can trust the provided information.
Within the talk, I will give an example how XAI technologies can be used in applications relating to critical infrastructure. Afterwards, I will have a look at challenges such as uniqueness and plausibility. I will present some results how to target these challenges: Specifically, I will have a closer look at feature-based explanations such as offered by SHAP as an approach rooted in an axiomatic treatment offered by game-theory. I will argue how effects offered by interactions can lead to different results and how feature relevance measures can be extended to quantify feature interactions, such as present in large language models or multimodal foundation models.
References:
- Fabian Fumagalli, Maximilian Muschalik, Eyke Hüllermeier, Barbara Hammer, Julia Herbinger:
Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory. AISTATS 2025: 5140-5148
- Maximilian Muschalik, Fabian Fumagalli, Paolo Frazzetto, Janine Strotherm, Luca Hermes, Alessandro Sperduti, Eyke Hüllermeier, Barbara Hammer:
Exact Computation of Any-Order Shapley Interactions for Graph Neural Networks. ICLR 2025
- Fabian Hinder, Valerie Vaquet, Barbara Hammer: One or two things we know about concept drift - a survey on monitoring in evolving environments. Part B: locating and explaining concept drift. Frontiers Artif. Intell. 7 (2024)
- Fabian Hinder, Valerie Vaquet, Barbara Hammer: Feature-based analyses of concept drift. Neurocomputing 600: 127968 (2024)
-Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, Eyke Hüllermeier: shapiq: Shapley Interactions for Machine Learning. NeurIPS 2024
Bio: Barbara Hammer chairs the Machine Learning research group at the Research Institute for Cognitive Interaction Technology (CITEC) at Bielefeld University. After completing her doctorate at the University of Osnabrück in 1999, she was Professor of Theoretical Computer Science at Clausthal University of Technology and a visiting researcher in Bangalore, Paris, Padua and Pisa. Her areas of specialisation include trustworthy AI, lifelong machine learning, and the combination of symbolic and sub-symbolic representations.
She is PI in the ERC Synergy Grant WaterFutures and in the DFG Transregio Contructing Explainability. Barbara Hammer has been active at IEEE CIS as member of chair of the Data Mining Technical committee and the Neural Networks Technical Committee. She has been elected as a review board member for Machine Learning of the German Research Foundation in 2024 and she represents computer science as a member of the selection committee for fellowships of the Alexander von Humboldt Foundation. She is member of the Scientific Directorate Schloss Dagstuhl. Further, she has been selected as member of Academia Europaea.
Dr. Jens Buß