Researchers have introduced ClosureBench, a new benchmark designed to evaluate compositional graph reasoning capabilities in AI models. Unlike traditional benchmarks, ClosureBench generates tasks on demand with programmatically verified answers, preventing data contamination and directly measuring memorization. Evaluations showed that model accuracy decreases with increased graph size and query depth, and that models struggle with compositional queries even when given structured graph inputs. Notably, a smaller model fine-tuned to generate executable programs achieved performance comparable to frontier models at a lower cost. AI
IMPACT Highlights limitations in current LLMs for complex reasoning and suggests program synthesis as a more efficient approach.
RANK_REASON The cluster contains a research paper introducing a new benchmark for AI model evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
