New QMFOL framework generates controllable logic reasoning benchmarks for LLMs

Researchers have introduced QMFOL, a novel framework designed to generate quantifiable and controllable monadic first-order logic reasoning tasks. This system addresses limitations in existing benchmarks by allowing precise control over logical complexity, semantic diversity, and consistency. The framework has been used to create QMFOLBench, a benchmark with 2880 instances, which has been used to evaluate six large reasoning models and two LLMs. The evaluations revealed that model performance decreases and computational demands increase with greater logical complexity, and that models are more accurate on 'True'-labeled tasks compared to 'False' or 'Unknown' ones. AI

IMPACT Provides a more precise method for evaluating LLM deductive reasoning, enabling better understanding of model limitations with increasing logical complexity.

RANK_REASON The cluster describes a new academic paper introducing a novel framework and benchmark for evaluating LLM reasoning capabilities.

Read on arXiv cs.AI →

• arXiv
• False
• Hugging Face
• large-language models
• LRMs
• QMFOL
• QMFOLBench
• True
• alphaXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →