Opus 5.5 and GPT-6 SOL show diminishing returns on coding benchmarks

A recent analysis of the SlopCodeBench benchmark suggests that increasing the effort level for models like Opus 5.5 and GPT-6 SOL may not yield significant improvements in strict pass rates, and can even lead to a decrease in performance for some models. The benchmark, which tests coding agents on long-horizon iterative tasks, found that while models often succeed at core functionality, they struggle to maintain previous features when extended. The analysis also highlights that the cost per successful pass increases substantially with higher effort settings, with GPT-6 Astra being a notable exception that showed improvement at a higher cost. AI

IMPACT Suggests that current high-effort settings for coding models may be inefficient, potentially guiding future model development and prompting strategies.

RANK_REASON Analysis of benchmark results for existing models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Dex Horthy
• Gabe Orlanski
• GPT-6 Astra
• GPT-6 SOL
• Hacker News
• HumanLayer
• Medium
• Opus 5.5
• SlopCodeBench
• Sonnet 5.5
• xHigh

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →