OpenAI 详细介绍了特定的 API 设置如何显著影响基准性能，特别是对其 GPT-5.6 "Sol" 模型。通过 Responses API 启用 "保留推理" 和 "上下文压缩"，GPT-5.6 Sol 在 ARC-AGI-3 基准测试中的分数提高了 188%，同时使用的输出 token 减少了六倍。OpenAI 认为，这些设置以及适当的框架设计和提示对于准确评估模型至关重要，尤其是在内存和上下文管理是关键的长期代理任务中。 AI
影响 强调了标准化评估实践和 API 配置对于准确衡量 AI 模型性能的重要性。
排序理由 OpenAI 正在解释其 API 设置如何影响基准结果，而不是宣布新模型或功能。
AI 生成摘要 · Google Gemini · 来自 5 个来源。 我们如何撰写摘要 →
We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize performance, we recommend using the
A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned. https://t.co/JBIKCKrUfw
We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens. https://t.co/uN1IrKEugu
ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over. https://t.co/cLtHYMNqiO
GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores ht…