A new benchmark called DeclineCraft has been developed to evaluate how well retrieval-augmented generation (RAG) systems admit when they cannot answer a question. The benchmark, which uses questions about Minecraft features beyond a specific date against a historical wiki corpus, found that while single-shot retrieval improves a system's ability to abstain from answering unanswerable questions, iterative retrieval and query rewriting can decrease this capability. The researchers emphasize that traditional evaluations focusing solely on accuracy for answerable questions can mask regressions in a system's AI
RANK_REASON [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
