vLLM preemption causes request restarts, increasing latency

The vLLM preemption issue occurs when the KV cache pool runs out of available blocks during request generation, leading the scheduler to evict in-progress requests. In the default RECOMPUTE mode, this discards the request's KV cache, forcing it to restart from scratch and causing significant latency spikes. This problem is often caused by admitting more concurrent sequences than the KV pool can sustain, especially with longer output lengths. Solutions include monitoring preemption counts, capping the maximum number of sequences, and setting token limits per request. AI

IMPACT This issue can significantly increase inference latency and costs for LLM deployments, impacting user experience and operational efficiency.

RANK_REASON The item discusses a specific technical issue and its solutions within the vLLM inference engine, which is a tool for serving LLMs.

Read on dev.to — LLM tag →

• Llama 3.1 8B-Instruct
• PagedAttention
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →