Brief

Researchers have developed the Robust Reasoning Benchmark (RRB), a new evaluation pipeline that tests large language models on mathematical problems with deliberate textual perturbations. The benchmark revealed that while frontier models are largely resilient, Anthropic's Claude model categorically refuses many transformed prompts. Open-weights models showed significant accuracy drops, with some experiencing up to a 54% decrease across various failure modes. The study also identified "Intra-Query Attention Dilution" as a key issue where intermediate reasoning steps degrade performance on subsequent problems within the same context window, suggesting a need for architectural changes to manage attention mechanisms. AI