A new benchmark called Physics has been developed to evaluate the capabilities of foundation models in solving advanced physics problems, including those found on PhD-qualifying exams. The benchmark comprises 1034 annotated problems across six physics domains, requiring both domain-specific knowledge and multi-step mathematical reasoning. Initial evaluations of 53 models showed that while some proprietary models and a few open-weight models achieved high accuracy on problems without figures, most open-weight models struggled, with Qwen3.5-9B exhibiting issues in knowledge application and figure interpretation. AI
IMPACT This benchmark could drive improvements in AI's reasoning and knowledge application capabilities, particularly in STEM fields.
RANK_REASON The cluster describes a new academic benchmark for evaluating AI models on a specific domain (physics problem-solving), presented in a research paper on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
