The Hugging Face SWE-bench Pro leaderboard, as of October 31, 2026, presents a complex picture with 47 rows, but the underlying benchmark has two distinct versions, V1 and V2. V1 contains 731 tasks, while V2, released on September 22, 2026, has 642 tasks after 89 were dropped and many others were repaired or retired. The top-ranked models on the leaderboard are not consistently using the same benchmark version or harness, leading to potential confusion in score interpretation. Concerns have also been raised about task integrity, potential data leakage from public repositories, and inconsistencies in reporting. AI
IMPACT Raises questions about the reliability of coding benchmarks, potentially impacting how model performance is evaluated and compared.
RANK_REASON Article discusses benchmark versions and scoring integrity, not a new model release. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
