Open Contrastive Decision Model Fails Public LLM Judge Benchmarks

A new research paper introduces Contrastive-LM/CLM-v0.1-8B, an open contrastive decision model designed for evaluating large language models. However, the model performed poorly on several public benchmarks, scoring near chance on RM-Bench and JudgeBench, and consistently labeling all items on HaluEval. While its raw confidences were overconfident, calibration improved its performance, though it remained statistically indistinguishable from random chance on key evaluations. The research also explored decision order-flip rates and length-preference shifts, finding CLM-v0.1-8B to be significantly better than a generative judge in these aspects. AI

IMPACT This research highlights the challenges in developing reliable LLM-based judges and suggests that current open contrastive models may not yet meet the performance standards of other evaluation methods.

RANK_REASON Research paper detailing the evaluation of a new LLM-based judge model. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Benjamini-Hochberg correction
• Contrastive-LM/CLM-v0.1-8B
• Gowtham Kumar Nanda Kishore
• HaluEval
• Hugging Face
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• RM-Bench

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →