PulseAugur
EN
LIVE 01:16:42

OpenAI reveals API settings boost GPT-5.6 Sol benchmark scores 188%

By PulseAugur Editorial · [5 sources] ·

OpenAI has detailed how specific API settings significantly impact benchmark performance, particularly for their GPT-5.6 "Sol" model. By enabling "retained reasoning" and "context compaction" through the Responses API, GPT-5.6 Sol saw an 188% score increase on the ARC-AGI-3 benchmark while using six times fewer output tokens. OpenAI suggests these settings, along with proper harness design and prompting, are crucial for accurately evaluating models, especially for long-running agent tasks where memory and context management are key. AI

IMPACT Highlights the importance of standardized evaluation practices and API configurations for accurate AI model performance measurement.

RANK_REASON OpenAI is explaining how their own API settings affect benchmark results, rather than announcing a new model or capability.

Read on X — OpenAI →

AI-generated summary · Google Gemini · from 5 sources. How we write summaries →

OpenAI reveals API settings boost GPT-5.6 Sol benchmark scores 188%

COVERAGE [5]

  1. X — OpenAI TIER_1 English(EN) · OpenAI ·

    We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API setting

    We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize performance, we recommend using the

  2. X — OpenAI TIER_1 English(EN) · OpenAI ·

    A benchmark score reflects the model as well as the harness and settings used to run it.

    A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned. https://t.co/JBIKCKrUfw

  3. X — OpenAI TIER_1 English(EN) · OpenAI ·

    We implemented the harness with the Responses API and turned on:

    We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens. https://t.co/uN1IrKEugu

  4. X — OpenAI TIER_1 English(EN) · OpenAI ·

    ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions.

    ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over. https://t.co/cLtHYMNqiO

  5. X — OpenAI TIER_1 English(EN) · OpenAI ·

    GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games?

    GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores ht…

RELATED ENTITIES

RELATED TOPICS