GPT-4o and other multimodal models evaluated on computer vision tasks

A new paper evaluates how well multimodal foundation models, including GPT-4o and Gemini 1.5 Pro, perform on standard computer vision tasks. Researchers developed a prompt-chaining method to translate vision tasks into text-based formats for API-accessible models. The study found that while these models are respectable generalists, they do not yet match specialized computer vision models, performing better on semantic than geometric tasks. GPT-4o showed the strongest performance among non-reasoning models, though models with native image generation capabilities exhibited failure modes like hallucinated objects. AI

IMPACT Assesses current multimodal model capabilities on vision tasks, highlighting limitations compared to specialized models.

RANK_REASON This is a research paper evaluating existing multimodal models on computer vision tasks.

Read on arXiv cs.CV →

• Claude 3.5 Sonnet
• COCO
• Gemini 1.5 Pro
• GPT-4o
• ImageNet
• Llama 3.2
• OpenAI
• Qwen2-VL

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →