OpenAI has demonstrated that a single model, GPT-5.6 "Sol", can achieve significantly improved performance on the ARC-AGI-3 benchmark by adjusting API settings rather than altering the model itself. By retaining the model's reasoning and compacting history instead of discarding it, the model's score increased from 13.3% to 38.3% while using one-sixth the output tokens. This suggests that previous benchmark results may have been artificially lowered due to configuration choices that induced a form of "anterograde amnesia" in the model, forcing it to re-derive solutions turn by turn. AI
IMPACT Highlights the critical role of configuration and prompt engineering in LLM performance, suggesting many benchmarks may be flawed.
RANK_REASON The item details a novel benchmark result and analysis of model configuration, not a direct model release. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
