LLM inference at temperature=0, which typically selects the highest probability token, is not guaranteed to be deterministic due to the nature of floating-point arithmetic and batching on GPUs. Concurrent requests sharing GPU resources can alter the order of operations in calculations like matrix multiplication and attention, leading to tiny variations in output probabilities. When the probabilities of the top two tokens are very close, these minor variations can cause the model to select a different token, resulting in a completely different output sequence. To achieve true bit-identical reproducibility, specialized batch-invariant kernels are required, which can incur a performance cost. AI
IMPACT Highlights a subtle but important detail for developers using LLMs for evaluation and regression testing.
RANK_REASON Technical explanation of a nuance in LLM inference behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
