New benchmark DNBENCH reveals LLM failures in database normalization

Researchers have developed a new benchmark called DNBENCH to evaluate the capabilities of Large Language Models (LLMs) in database normalization tasks. This benchmark includes 3,275 samples designed to test LLMs' ability to reason about functional dependencies and constraints, from first normal form (1NF) up to Boyce-Codd normal form (BCNF). The study revealed recurring failures in LLMs' dependency inference and schema decomposition. To address these issues, a multi-agent framework named MARS was proposed, which significantly improved normalization scores by separating different reasoning stages. AI

IMPACT Highlights limitations of current LLMs in structured data tasks, potentially guiding future research in robust database interaction.

RANK_REASON The cluster contains an academic paper detailing a new benchmark and framework for evaluating LLM capabilities in database normalization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Boyce–Codd normal form
• DNBENCH
• first normal form
• LLMs
• MARS

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →