Scaffold Effects on GAIA: A Controlled Comparison
A new study published on arXiv reveals that the way AI models are prompted, or "scaffolded," significantly impacts their measured performance. Researchers found that the choice of scaffold alone could alter a model's accuracy by up to 28 percentage points. Contrary to expectations, more capable models were not necessarily less sensitive to scaffolding, with some advanced models showing greater gains from structured prompts. The findings suggest that current capability scores may be overly dependent on the specific prompting methods used, rather than solely reflecting inherent model abilities. AI
IMPACT Highlights the critical role of prompting techniques in evaluating AI capabilities, suggesting current benchmarks may not fully capture true model potential.