Researchers have introduced ARCS, a new benchmark designed to evaluate text-to-SQL systems' ability to handle ambiguity in user queries. Traditional methods of resolving ambiguity through conversation are inefficient; ARCS proposes a structured disambiguation approach. The benchmark includes naturally occurring ambiguities over real-world databases, with detailed annotations. Initial experiments show that current models struggle with ambiguity, with GPT-6 Sol achieving only 51% execution accuracy and open-source models performing even worse. AI
IMPACT Highlights a critical challenge for text-to-SQL systems, potentially driving research into more robust ambiguity resolution techniques.
RANK_REASON The cluster describes a new academic benchmark and associated dataset for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →