New LGMT tool reveals significant logical reasoning inconsistencies in LLMs

A new open-source tool called LGMT has been developed to test the logical reasoning consistency of Large Language Models (LLMs). LGMT identifies inconsistencies by generating logically equivalent restatements of queries and checking if the LLM provides consistent answers. In tests across six LLMs, LGMT revealed inconsistency rates between 20.84% and 29.44%. The tool is available on GitHub and includes 20 built-in metamorphic relations, with support for custom relations verified by the Z3 theorem prover. AI

IMPACT Highlights a critical gap in LLM reasoning evaluation, potentially driving development of more robust and reliable models for logical tasks.

RANK_REASON The cluster describes a new research paper detailing an open-source tool for evaluating LLM reasoning capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• first-order logic
• GitHub
• Hugging Face
• Large Language Models
• Z3 Theorem Prover

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →