Researchers have developed a new benchmark, the Controlled Attribution Benchmark (CATT), to systematically evaluate data attribution methods for language models. This benchmark uses entirely synthetic data, allowing for precise control over dataset properties like redundancy and distractors, which were found to degrade attribution performance. The study also identified that averaging attribution scores across entire documents dilutes the signal from critical tokens. By replacing standard cross-entropy loss gradients with negative log-odds gradients and aggregating only top token scores, performance improved significantly, demonstrating the importance of localized evidence in attribution. AI
IMPACT Introduces a new diagnostic tool for understanding and improving language model data attribution, potentially leading to more reliable and interpretable AI systems.
RANK_REASON The cluster describes a new benchmark and methodology for evaluating language model data attribution, presented in a research paper. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
