Kish

LLM evaluation confidence intervals are too narrow due to clustered data

A common assumption in evaluating language models is that each test example is independent, but this is often violated in practice. When examples are clustered (e.g., multiple questions from the same document or turns f…