LLM KV cache, not weights, drives VRAM needs for long contexts

The memory requirements for running large language models (LLMs) extend beyond just model weights, with the KV cache being a significant factor, especially for long context windows. For instance, a 7B parameter model like Llama 3.1 8B can require approximately 128 KiB per token for its KV cache at a 128K context length, drastically increasing VRAM needs. Architectural differences, such as KV head count, also impact cache size, with Qwen2.5 7B needing less cache than Llama 3.1 8B for the same context. Deploying larger models like Llama 3.1 70B with long contexts necessitates multi-GPU setups, even with FP8 quantization, due to the combined size of weights and KV cache. Factors like batch size, the efficiency of memory management techniques like PagedAttention, and KV cache quantization are crucial for accurate VRAM budgeting. AI

IMPACT Understanding KV cache requirements is crucial for optimizing LLM deployment costs and performance, especially with increasing context window sizes.

RANK_REASON The item is a technical explanation and analysis of LLM inference costs, not a primary release or significant industry event.

Read on dev.to — LLM tag →

• FP8
• FP16
• KV cache
• Llama 3.1 70B
• Llama 3.1 8B
• PagedAttention
• Qwen2.5 7B
• RTX 4090
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →