New ASyMOB benchmark tests LLM math reasoning beyond memorization

Researchers have introduced ASyMOB, a new benchmark designed to evaluate the symbolic mathematics capabilities of large language models. The dataset contains over 35,000 validated problems across various mathematical domains, with a focus on testing generalization through symbolic and numeric transformations. Initial evaluations show that most models struggle with minor perturbations, though top systems demonstrate improved robustness, and the integration of code tools significantly stabilizes performance. AI

IMPACT Provides a more rigorous evaluation for LLMs in symbolic mathematics, pushing development towards genuine reasoning over memorization.

RANK_REASON New academic paper introducing a benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• ASyMOB
• Ido Kaminer

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →