arxiv:2609.04198v1

LLM benchmarks fail reliability tests, study finds

A new study reveals significant unreliability in current LLM-based evaluation benchmarks, even when using identical inputs and zero temperature settings. Researchers found that rerunning the same agent outputs through s…