Researchers have introduced ResearchQA, a new benchmark designed to evaluate how well large language models can answer questions based on scientific papers while ensuring answers are supported by verifiable citations. The benchmark includes over 6,000 question-answer pairs across various domains and question types, specifically rewarding grounded refusal when a paper does not support an answer. Evaluations of eight leading models showed that citation-based metrics were more effective at differentiating model performance than LLM-based rubric evaluations, with open-weight models demonstrating competitive citation accuracy and significantly lower latency compared to closed-weight models. AI
IMPACT This benchmark could drive improvements in LLM accuracy and reliability for scientific research and knowledge extraction.
RANK_REASON The cluster describes the release of a new academic benchmark for evaluating LLMs on a specific task.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
