LLM-as-Judge systems conflate trust and truth, study finds

A new research paper explores the separation between trust and truth judgments made by Large Language Models (LLMs) when used as judges. The study found that LLM judges tend to conflate trust scores with truthfulness more than humans do, suggesting these judgments are not as independent as often assumed. When the source of information was manipulated, LLM judges altered both their trust scores and truth verdicts, indicating a susceptibility to external cues rather than purely factual assessment. AI

IMPACT Findings suggest current LLM-as-Judge protocols may require refinement to ensure independent evaluation of truthfulness.

RANK_REASON Research paper published on arXiv detailing findings about LLM-as-Judge systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• human
• LLM-as-Judge
• QA

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →