LLM-as-a-Judge: Unpacking Evaluation Mechanisms in Summarization

Researchers have investigated the internal mechanisms of Large Language Models (LLMs) when used as judges for evaluating natural language generation quality. By applying causal tracing and other analysis techniques to models like Themis (Llama-3-8B) and Prometheus (Mistral-7B), they found that these LLMs follow a structured evaluation pipeline. This pipeline involves attention mechanisms in lower layers for local error comparison and MLP cascades in higher layers for integrating signals and assigning ratings, with the decision solidifying in the residual stream at late layers. AI

影响 Provides insight into how LLMs function as evaluators, potentially improving future NLG quality assessment and training signal generation.

排序理由 The cluster contains an academic paper detailing research into LLM evaluation mechanisms. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• Llama 3-8B
• Mistral-7B
• Prometheus
• ScienceCast
• Themis

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →