New benchmark aims to align LLM survey evaluators with human reviewers

Researchers have introduced SurveyReview, a new benchmark and dataset designed to evaluate large language models (LLMs) when they are used as survey evaluators. This benchmark addresses the lack of systematic alignment with human reviewers in existing LLM-as-a-judge methods. SurveyReview includes 675 annotated survey papers with 1,630 review reports, structured into four-dimensional scores and rationales. A baseline evaluator, SurveyAlign, fine-tuned on Qwen3-32B, demonstrated significant improvements in reviewer alignment compared to GPT-5.2, reducing mean squared error and mean absolute error. AI

IMPACT This benchmark could lead to more reliable LLM-based evaluation of survey research, improving the quality and efficiency of academic peer review processes.

RANK_REASON The item describes a new academic paper introducing a benchmark and dataset for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• DagsHub
• GPT-5.2
• Hugging Face
• LoRA+
• Qwen3 32B
• SurveyAlign
• SurveyReview

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →