New benchmark tests LLMs' ability to formalize geometry problems for AlphaGeometry

Researchers have developed NL2AGBench, a new benchmark designed to evaluate how well large language models can translate informal geometry problems into the formal language required by AlphaGeometry. This is crucial because AlphaGeometry, which performs at a near-gold medalist level in the International Mathematical Olympiad, requires inputs in a specialized domain-specific language, and manual conversion is a significant bottleneck. The benchmark uses execution-based verification within AlphaGeometry to assess translation quality. Experiments showed that leading closed-source LLMs achieved over 80% executable translation rates, significantly outperforming open-source models, which struggled to produce valid formalizations. AI

IMPACT This benchmark could accelerate the development of LLMs capable of formalizing complex mathematical problems, potentially aiding in automated theorem proving and scientific discovery.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLM capabilities in a specific domain. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AlphaGeometry
• arXiv
• domain-specific language
• Hugging Face
• International Mathematical Olympiad
• Large language models
• NL2AGBench

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →