LLM Decode Speed: Bandwidth and KV Cache Limit Performance on H100

A technical analysis reveals that for large language models running on hardware like the NVIDIA H100, decode speed at batch size one is primarily determined by memory bandwidth rather than compute power. The "ridge batch" calculation shows that as context length increases, the KV cache becomes a bottleneck, making models memory-bound even with batching. This limits the practical token generation rate, with specific ceilings calculated for models like Qwen3-32B and Llama-3.3-70B based on their weights and KV cache requirements. AI

IMPACT Highlights memory bandwidth and KV cache as key constraints for LLM inference performance, impacting deployment strategies.

RANK_REASON Technical analysis of LLM performance bottlenecks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• 3.35 TB/s
• 80 GiB
• bfloat16
• H100 SXM
• HBM3
• Llama-3.1:8b
• Llama 3.3 70B Instruct
• NVIDIA H100
• Paul Irolla
• Qwen3 32B

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →