A new preprint from Peking University evaluated the ability of large language models to reproduce numerical results from experimental physics papers. Researchers found that all tested LLMs, including OpenAI Codex powered by GPT-5.3, achieved a 0% end-to-end callback rate, meaning they could not replicate any full numerical outcomes. While the models demonstrated strong comprehension of the papers' methodologies, they consistently made errors in data analysis and numerical simulation, leading to incorrect final results. The study identified several failure modes, such as formula implementation errors and oversimplification of complex physical models. AI
Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →
IMPACT LLMs struggle with complex numerical simulation and data analysis in scientific research, indicating limitations beyond text comprehension.
RANK_REASON Academic paper evaluating LLM capabilities on a new domain (physics simulation).