New SAGE-QA framework improves LLM evaluation on SQuAD 2.0 benchmark

A new evaluation framework called SAGE-QA has been introduced to better assess the performance of generative large language models on closed-context question answering tasks, specifically for the SQuAD 2.0 benchmark. Traditional Exact Match (EM) scoring can be misleading for LLMs, as it penalizes valid paraphrases. SAGE-QA combines semantic equivalence scoring and passage-based validation to differentiate factual errors from lexical variations. In zero-shot evaluations, Claude Opus-4.6 and Gemini-3.1 Pro achieved high EM scores, but fine-tuned open-weight models like Llama 3.3-70B surpassed them, demonstrating the effectiveness of parameter-efficient fine-tuning. AI

IMPACT Introduces a more robust evaluation method for LLMs, potentially leading to more accurate assessments of generative QA capabilities.

RANK_REASON The item is an academic paper introducing a new evaluation framework for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Claude Opus-4.6
• Exact Match (EM)
• Gemini-3.1 Pro
• GPT OSS 20B
• Llama 3.3-70B
• Llama 3-8B
• Re-Validation Scoring (RVS)
• SAGE-QA
• Semantic Equivalence Scoring (SES)

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →