New 'Representation-as-a-Judge' method uses small models for evaluation

Researchers have proposed a new evaluation method for language models called Representation-as-a-Judge, which utilizes the internal representations of smaller models rather than their generative output. This approach is based on the Semantic Capacity Asymmetry Hypothesis, suggesting that evaluation requires less semantic capacity than generation. The proposed framework, INSPECTOR, leverages these internal features from small models to predict evaluation scores, offering a more efficient, reliable, and interpretable alternative to traditional LLM-as-a-Judge methods. Experiments on reasoning benchmarks like GSM8K, MATH, and GPQA demonstrate that INSPECTOR performs comparably to larger LLM judges while being significantly more efficient. AI

IMPACT This research could lead to more efficient and interpretable AI evaluation systems, reducing reliance on large, costly models.

RANK_REASON The cluster contains a research paper detailing a new methodology for evaluating language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• GSM8K
• INSPECTOR
• LLM-as-a-Judge
• Representation-as-a-Judge
• Semantic Capacity Asymmetry Hypothesis
• Zhuochun Li

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →