Expert vs. Crowd Annotator Disagreement Masks Model Performance Variability

A recent study published on arXiv highlights significant disagreements between expert and crowd annotators when evaluating language models. Despite these substantial differences in item-level judgments, the resulting leaderboards for models remain largely identical, suggesting that current aggregation methods obscure underlying variability. The research indicates that the choice of annotator pool can influence a model's perceived performance, and that LLM judges tend to align more with crowd annotators than experts. AI

IMPACT Highlights potential biases in AI model evaluation and the need for more robust benchmarking methodologies.

RANK_REASON The cluster contains a research paper published on arXiv discussing methodology in AI model evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• DagsHub
• Gotit.pub
• Hugging Face
• MT-Bench
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →