Researchers have introduced ReqClarifyBench, a new benchmark designed to evaluate the interactive requirement clarification capabilities of large language models (LLMs) in repository-level software development. The benchmark includes 341 tasks across various development types, each starting with a vague user intent and progressing to a human-validated requirement document. Evaluations of six LLMs revealed that improved clarification effectiveness did not always correlate with higher-quality requirement documents, and performance varied significantly based on user personas. The study also found that coding-agent harnesses did not enhance requirement document quality beyond their base LLMs, offering insights for future end-to-end LLM-based software development systems. AI
IMPACT This benchmark could drive improvements in LLM-based software development tools by highlighting the challenges in requirement clarification.
RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
