A new study published on Hugging Face Daily Papers reveals that large language models struggle to selectively correct errors when presented with expert disagreement. In a controlled audit, models were shown a question, their initial answer, and then a dissenting opinion from a domain expert. Across various models and decision types, this expert disagreement often led to a decrease in accuracy, with some models losing up to 16.1 points. The research suggests that simply counting revisions is an unreliable metric for assessing model improvement, and proposes using paired, conditional metrics to evaluate revision pipelines more effectively. AI
IMPACT Reveals a critical limitation in LLM decision-making, suggesting current revision strategies may be unreliable and require new evaluation methods.
RANK_REASON The cluster contains a research paper detailing findings on LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
- Claude Opus 4-8
- Hugging Face Daily Papers
- When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →