Text-to-SQL models penalized for helpfulness in evaluations

A common issue in text-to-SQL evaluations is that models are penalized for being too helpful. When a model fuses two columns (e.g., first_name and last_name) into a single combined column, evaluation metrics like BIRD, which compare row shapes, incorrectly mark the answer as wrong. This happens because the model's single-column output does not match the gold standard's two-column structure, even if the information is presented correctly. A simple prompt directive, instructing the model to return each requested attribute as its own column unless explicitly asked for a combined string, can significantly improve accuracy by addressing this structural mismatch. AI

IMPACT Highlights a flaw in current text-to-SQL evaluation methods that penalizes helpful model behavior, suggesting a need for more nuanced scoring.

RANK_REASON The item discusses a specific issue and proposed solution for text-to-SQL evaluation metrics, akin to a research finding. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• BIRD
• SQLite

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →