On Pitfalls of $\textit{RemOve-And-Retrain}$: Data Processing Inequality Perspective
A new paper from Junghoon Seo on arXiv explores the limitations of the RemOve-And-Retrain (ROAR) benchmark, commonly used to assess feature attribution methods. The research indicates that post-processing attribution maps, which cannot add information according to the data processing inequality, can artificially inflate ROAR scores. This suggests that improved ROAR rankings do not necessarily correlate with attribution maps containing more information about a model's decision-making process. Experiments on datasets like CIFAR-10 and SVHN reveal a tendency for blurrier masks to perform better, highlighting a potential bias in the benchmark. The authors propose guidelines for more reliable benchmarking to better understand neural network internals. AI
IMPACT Challenges the reliability of a common benchmark for AI feature attribution methods, potentially impacting how model interpretability is evaluated.