New benchmark evaluates LLMs' ability to find errors in synthetic math data

Researchers have introduced MathDebugger, a new benchmark designed to evaluate the ability of large language models to detect and diagnose errors within synthetic mathematical data. The benchmark includes a dataset of correct and erroneous questions, along with annotated solutions, achieving high agreement among human annotators. Evaluations of 14 large language models and three process reward models revealed that even advanced models struggle to fully saturate MathDebugger, particularly with fine-grained error identification. The study also highlighted a gap between models' solving and verification capabilities, suggesting that explicit error information can guide improvements in data synthesis and quality control pipelines. AI

IMPACT This benchmark could drive improvements in the reliability of synthetic data used for training LLMs, leading to more robust reasoning capabilities.

RANK_REASON The cluster is about an academic paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Fleiss' kappa
• Gotit.pub
• Hao Liang
• Hugging Face
• large language models
• MathDebugger
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →