Large language models (LLMs) exhibit non-reproducible outputs even when temperature is set to 0, due to batching during inference. This occurs because batched computations involve different matrix multiplications and floating-point arithmetic than unbatched ones, leading to fractional differences in logits that can alter token selection when probabilities are close. The author discovered this issue when an A/B test comparing a filter's effectiveness yielded an impossible result, revealing that the underlying model outputs were significantly different (around 30% of records changed) between runs, not due to sampling but due to batching. AI
IMPACT LLM output reproducibility is critical for research and production; this finding suggests that batching can introduce subtle but significant variations, impacting benchmarks and A/B testing.
RANK_REASON The item details a technical finding about LLM inference behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
