New ARCANA benchmark reveals analogical reasoning gap in LLMs

Researchers have introduced ARCANA, a new benchmark designed to evaluate far analogical reasoning in large language models. Unlike the ARC-AGI benchmark, ARCANA focuses on mapping real-world phenomena to abstract transformations, a task humans perform reliably but current LLMs struggle with. To bridge this performance gap, a framework called ConceptSIR was proposed, which incorporates concept slippage, introspection, and reflection, showing improved results on challenging ARCANA tasks. AI

IMPACT Highlights a significant gap in LLM capabilities for analogical reasoning, suggesting current models struggle with abstract mapping tasks.

RANK_REASON The cluster describes a new academic benchmark and a proposed framework for evaluating LLM capabilities, published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Abstraction and Reasoning Corpus of ANAlogy
• ARC-AGI
• ARCANA
• arXiv
• ConceptSIR
• large language models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →