LLM temperature=0 outputs can differ due to floating-point math and shared GPUs

The common assumption that setting `temperature=0` in large language models guarantees deterministic output is flawed. This is because floating-point arithmetic, specifically in `float16` precision, is not associative, leading to slightly different results when sums are computed in different orders. Furthermore, GPU kernels used for matrix multiplications in LLM inference are not always deterministic in their summation order, as this can depend on factors like batch size, which is influenced by the traffic of other users sharing the GPU. These subtle variations in computed logits can cause the model to select a different token, leading to entirely different generated text, even with the same prompt and parameters. AI

IMPACT Highlights a subtle but critical issue for reproducible LLM outputs, impacting testing and deployment.

RANK_REASON Explains a technical nuance of LLM inference related to floating-point arithmetic and GPU execution. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• GPU kernels
• graphics processing unit
• half-precision floating-point format
• NumPy
• temperature=0

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →