Language models prove 30% of math conjectures in new benchmark

Researchers have developed OEIS Open, a new benchmark designed to evaluate how many mathematical conjectures language models can prove. This benchmark, based on 492 open conjectures from the On-Line Encyclopedia of Integer Sequences formalized in Lean, allows any generic language model to be tested. Initial results show that LMs can resolve a significant portion of these conjectures autonomously, with one model scoring 44% on a subset of the benchmark using a budget of $200 per attempt. AI

IMPACT Demonstrates potential for LLMs to autonomously discover mathematical proofs, potentially accelerating research in formal mathematics.

RANK_REASON The cluster describes a new benchmark for evaluating language models on mathematical conjectures, based on an academic paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Language Models
• Lean
• OEIS Open
• On-Line Encyclopedia of Integer Sequences
• Thomas Adamczewski
• Tsoukalas et al.

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →