OpenAI's o4-mini model achieves 90% accuracy on physics problems, but struggles with images

A new arXiv paper evaluates OpenAI's o4-mini model on introductory physics problems from Halliday and Resnick's "Fundamentals of Physics." The model achieved approximately 90% accuracy overall, but performance varied significantly based on problem representation and difficulty. Text-only problems were solved with 96% accuracy, while problems requiring interpretation of both text and images saw only 79% accuracy. The model's accuracy also decreased notably as problem difficulty increased. AI

IMPACT Demonstrates current LLM capabilities in complex problem-solving but highlights limitations in multimodal understanding and difficulty scaling.

RANK_REASON Academic paper evaluating an AI model's performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Halliday and Resnick's "Fundamentals of Physics"
• o4-mini
• OpenAI

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →