A new benchmark called Choice2Seq has been developed to evaluate how well AI models retain correctness when constructing answers through a sequence of local decisions, rather than just selecting a final answer. The benchmark revealed significant discrepancies between models' selection accuracy and their ability to construct correct answers sequentially. For instance, Jev achieved high selection accuracy on arithmetic and lexical retrieval tasks but retained only a fraction of those correct choices during serial construction. Similar gaps were observed in other models like StartLux and Qwen, even when they emitted characters directly. The research suggests that interventions such as restricting local options or using explicit position indexing can improve construction fidelity and distinguish between task correctness, plan fidelity, and output compliance. AI
IMPACT Highlights a critical gap in current LLM evaluation, suggesting future models need to improve sequential reasoning and output construction.
RANK_REASON Academic paper introducing a new benchmark. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
