A recent regression test revealed that vLLM's temperature 0 setting, intended for deterministic output, is not consistently producing the same results. This non-determinism arises because vLLM uses continuous batching, meaning a request's batch composition can change based on other concurrent requests sharing a GPU. The differing batch sizes lead to GPU kernels summing floating-point numbers in a different order, resulting in slightly altered logits. When the probabilities for the top two tokens are very close, this can cause the model to select a different token, which then cascades into divergent outputs for the remainder of the generated text. The issue was observed with Llama 3.1 8B Instruct and highlighted the impact of batch composition on inference results. AI
IMPACT Highlights potential issues in reproducible LLM evaluations and the impact of hardware batching on model output consistency.
RANK_REASON Technical analysis of an LLM inference system's behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
