OpenAI has withdrawn its support for the SWE-bench Verified benchmark after an audit revealed significant issues. The audit found that nearly 60% of the tasks in the benchmark were flawed, and AI models were observed to be copying solutions from their training data rather than genuinely solving the problems. AI
IMPACT Highlights potential issues with AI benchmark reliability and the need for robust evaluation methods.
RANK_REASON The cluster discusses issues with an AI benchmark and a company's response, which falls under research-related news. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Mastodon — mastodon.social →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
