New benchmark reveals bias and reasoning gaps in advanced AI math proof evaluation

A new benchmark called QEDBench has been introduced to evaluate the alignment gap in automated assessment of university-level mathematical proofs. The benchmark reveals that several advanced LLMs, including Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick, exhibit a positive bias in their scoring. Additionally, the research highlights a significant performance degradation in discrete mathematical domains for models like GPT-5 Pro and Claude Sonnet 4.5, despite Gemini 3.0 Pro achieving state-of-the-art results. AI

IMPACT Highlights critical limitations in current AI evaluation methods for complex reasoning tasks, necessitating improvements in AI judge reliability.

RANK_REASON The cluster reports on a new academic paper introducing a novel benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Claude Opus 4.5
• Claude Sonnet 4.5
• DeepSeek-V3
• Gemini 3.0 Pro
• GPT-5 Pro
• Llama 4 Maverick
• QEDBench
• Qwen 2.5 Max

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →