LLM requests split into compute-heavy prefill and VRAM-bound decode phases

LLM requests are processed in two distinct phases, each with its own bottleneck. The first phase, 'prefill,' involves the model processing the entire user prompt simultaneously, which is compute-heavy and limited by the GPU cores. This phase generates the initial token and a KV cache, a set of working notes saved in VRAM. The second phase, 'decode,' generates the rest of the answer word by word. In this phase, the bottleneck shifts to VRAM and data transfer, as the entire model weights must be read for each word generated, leaving the GPU cores largely underutilized. AI

IMPACT Understanding LLM request processing can help optimize inference performance and resource utilization.

RANK_REASON The item explains a technical concept about LLM processing rather than announcing a new product or research.

Read on dev.to — LLM tag →

• ChatGPT
• graphics processing unit
• KV cache
• VRAM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →