A new paper published on arXiv, titled "Averaging Bias: Human Faithfulness Annotations are not Locally Faithful," investigates the reliability of human annotations for evaluating the faithfulness of text summarization models. The study found that human annotators may exhibit an "Averaging Bias," where they label summaries as faithful even if they contain factual errors, rather than strictly adhering to a rule that requires every sentence to be supported by the source document. This bias was identified by comparing global human labels with per-sentence judgments from LLM judges across several benchmarks, suggesting a need for improved annotation designs to ensure trustworthy evaluations. AI
IMPACT Highlights potential flaws in current AI evaluation methods, suggesting a need for more rigorous human annotation designs for faithfulness.
RANK_REASON The cluster contains a single academic paper published on arXiv discussing a novel concept related to AI evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
