New research questions validity of AI feature attribution benchmark

A new paper from Junghoon Seo on arXiv explores the limitations of the RemOve-And-Retrain (ROAR) benchmark, commonly used to assess feature attribution methods. The research indicates that post-processing attribution maps, which cannot add information according to the data processing inequality, can artificially inflate ROAR scores. This suggests that improved ROAR rankings do not necessarily correlate with attribution maps containing more information about a model's decision-making process. Experiments on datasets like CIFAR-10 and SVHN reveal a tendency for blurrier masks to perform better, highlighting a potential bias in the benchmark. The authors propose guidelines for more reliable benchmarking to better understand neural network internals. AI

IMPACT Challenges the reliability of a common benchmark for AI feature attribution methods, potentially impacting how model interpretability is evaluated.

RANK_REASON Academic paper published on arXiv discussing a benchmark's limitations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• CIFAR-10
• Cub 200 2011 Caltech Birds Dataset
• Junghoon Seo
• RemOve-And-Retrain
• The Road
• The Street View House Numbers Dataset

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →