New benchmark reveals LLMs struggle with complex logic instructions

A new research paper introduces LogicIFGen and LogicIFEval, a framework and benchmark designed to evaluate Large Language Models' (LLMs) ability to follow complex logic instructions. The LogicIFGen framework automates the generation of verifiable instructions from code functions, incorporating intricate logic such as conditions, loops, and function calls. Experiments using the curated LogicIFEval benchmark, which contains 426 such instructions, reveal that current state-of-the-art LLMs struggle significantly, correctly following fewer than 60% of the logic-rich instructions. This highlights a notable deficiency in the instruction-following capabilities of existing LLMs when faced with complex logical structures. AI

IMPACT Highlights significant limitations in LLM reasoning and instruction-following for complex logical tasks, indicating areas for future model development.

RANK_REASON Research paper introducing a new benchmark and framework for evaluating LLM instruction following. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• LogicIF
• LogicIFEval
• LogicIFGen
• Mian Zhang
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →